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SECTION  I 
INTRODUCTION 

Self-paced  instruction  in  the  Navy  is  being  criticized  for  producing 
graduates  who  have  not  retained  the  information  taught  and  who  cannot 
perform  on  the  job.l  In  the  face  of  such  criticism,  some  self-paced  courses 
may  be  converted  back  to  conventional  group-paced  instruction. 

Unfortunately,  there  is  widespread  misconception  in  the  Navy  about 
self-paced  instruction.  It  supposes  that  the  huge  differences  in  student 
abilities  can  somehow  be  accommodated  by  merely  allowing  the  time  for 
learning  to  vary.  It  also  supposes  that  when  you  call  a  course  "self-paced" 
it  is  equivalent  to  individualized  instruction  (II).  In  fact, 
individualized  instruction  requires  the  use  of  a  number  of  sound 
instructional  elements;  self-pacing  is  only  one  of  these  elements. 

Self-paced  instruction,  .  with  some  additional  individualization  of 
instruction,  was  implemented  in  the  Navy  to  provide  effective  instruction  at 
lower  cost.  This  endeavor  has  lead  to  a  proliferation  of  nearly  200  such 
courses  affecting  nearly  100,000  students  yearly.  These  attempts  to  modify 
strategy  have  yielded  a  number  of  ways  in  which  courses  are  taught.  Some  of 
these  attempts  have  been  highly  successful;  others  are  criticized  as 
ineffective.  Disagreement  over  the  effectiveness  of  a  given  method  of 
delivery  of  instruction  may  be  due  to  the  various  meanings  attributed  to 
"self-paced"  instruction. 

Some  time  ago,  The  Training  Analysis  and  Evaluation  Group  (TAEG) 
pointed  out  this  confusion  in  terminology  and  recommended  a  comprehensive 
survey  to  establish  the  types  and  extent  of  II  in  use  in  the  Navy 
(Zajkowski,  Heidt,  Corey,  Mew,  and  Micheli,  1979).  Later,  Hall  and  Freda 
(1982)  suggested  that  there  are  apparently  many  meanings  to  II,  and  that  it 
is  not  a  unitary  concept.  Both  reports  show  there  is  a  need  to  identify  and 
to  differentiate  the  variety  of  instructional  practices  now  categorized  as 
II  in  the  Naval  Education  and  Training  Command  (NAVEDTRACOM) .  Consequently, 
the  Chief  of  Naval  Education  and  Training  (CNET)  tasked  the  TAEG  to  conduct 
a  study  of  the  variations  in  this  type  of  instruction. 2 

PURPOSE  OF  THE  STUDY 

The  purpose  of  the  present  study  is  to  identify  and  document  the  range 
of  instructional  activities  present  in  courses  thought  to  be 
"individualized"  by  the  NAVEDTRACOM.  A  model  of  individualized  instruction 
is  developed  to  provide  a  basis  for  qualitative  assessment  and  as  a  guide 
for  development  of  courses.  Proposed  changes  to  existing  directives  are 
provided  to  guide  course  designers  in  creating  appropriate,  efficient,  and 
effective  instruction. 

^-Commander  Patrol  Wings  Pacific  Itr  ser  70/1065  of  31  August  1982. 

2CNET  Code  022  Itr  to  CNTT  of  5  April  1982. 
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DEFINITIONS 

Three  categories  of  instruction  characterize  the  treatment  that  will 
follow.  They  are:  conventional,  mixed,  and  self-paced  instruction. 
Individualized  instruction  can  occur  within  these  categories  by  degrees. 

CONVENTIONAL  INSTRUCTION  (Cl).  The  central  features  of  Cl  include  group 
pacing,  lectures,  students  selected  with  similar  academic  aptitudes,  and  a 
single  form  of  instructional  material.  Conventional  instruction  appears  to 
prohibit  features  that  would  individualize  the  instruction;  however,  many 
instructors  have  developed  subtle  ways  of  individualizing  within  the 
constraints  of  this  seemingly  invariant  model. 

MIXED  INSTRUCTION  (MIX).  The  term  "mixed  instruction"  is  operationally 
defined  to  represent  courses  reporting  between  5  and  90  percent  "seif¬ 
pacing"  in  the  preliminary  survey  conducted  for  this  study. 

SELF-PACED  INSTRUCTION  (SP).  This  type  of  instruction  allows  students  of 
different  aptitudes  and  previous  knowledge  to  progress  through  a  program  at 
their  own  rates.  While  not  inherent  in  the  definition,  many  assume  the  term 
"self-paced"  instruction  includes  many  of  the  other  characteristics  of 
individualized  instruction  in  addition  to  releasing  the  student  from  time 
constraints.  This  study  does  not  make  such  assumptions. 

INDIVIDUALIZED  INSTRUCTION  (II).  This  term  is  defined  as  instructional 
activity  designed  to  accommodate  individual  human  differences  in  background, 
skill  level,  aptitudes,  and  learning  styles  characterized  by  (1)  releasing 
of  time  constraints,  (2)  choice  of  instructional  media,  and  (3)  adjustment 
to  skill  levels  and  learner  characteristics  (Zajkowski,  et  al.,  1979). 
Additionally,  those  operations  in  traditional  classrooms,  self-paced 
learning  centers,  and  mixed  environments  that  bring  about  the 
individualizing  of  the  instruction  are  identified.  These  operations  are 
further  defined  in  the  following  paragraphs. 

QUALITY  OF  INSTRUCTION  (QI) .  Six  elements  of  quality  instruction  will  be 
used  in  studying  all  of  the  courses  selected  in  the  sample  used  in  the 
present  .  study.  They  are  based  on  the  work  of  Bloom  (1976)  and  his 
colleagues  who  have  developed  a  system  of  Learning  for  Mastery  (Bloom, 
1968).  Here,  courses  are  assessed  in  terms  of  the  extent  to  which  the 
following  six  elements  (identified  by  Bloom)  are  present: 

1.  Prerequisites  (PRO).  These  are  the  cognitive  entry  behaviors  that 
are  measured  by  the  Armed  Services  Vocational  Aptitude  Battery  reading  and 
computational  scores,  or  similar  achievement  tests.  Other  achievement  tests 
show  the  students'  readiness  for  learning  a  particular  lesson  or  module  of 
instruction.  Affective  entry  behaviors  are  attitudes  reflected  in  measures 
of  motivation  and  perseverance.  The  most  adaptive  instruction  accommodates 
student  variation  in  both  the  cognitive  and  affective  entry  behaviors  in 
deciding  specific  instruction  for  a  given  student. 
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2.  Cues  (CUE).  These  are  the  signals  telling  the  students  what  it  is 
they  must  learn.  They  are  a  major  part  of  the  instructional  materials 
package,  as  well  as  the  on-the-block  instructor's  lecture.  Learning 
objectives,  the  format  of  printed  materials,  the  rubrics  and  headings, 
graphics,  diagrams,  mnemonics,  demonstrations,  topic  sentences,  and  verbal 
prompts  are  but  some  of  the  cues  that  bombard  a  student  in  the  typical 
classroom,  learning  center,  or  laboratory. 

3.  Participation  (PAR).  This  is  the  extent  to  which  students  are 
given  opportunities  to  practice  and  rehearse  that  which  they  are  to  learn. 
It  includes  exercise  materials  immediately  following  a  small  segment  of 
activity  and  the  distributed  practice  of  this  activity  over  a  period  of 
time.  To  learn  something,  the  student  must  do  something.  This  "doing"  may 
be  either  in  the  form  of  drill,  exercises,  or  quietly  thinking  about 
something  related  to  the  module  of  instruction.  Since  there  is  a  very  high 
relationship  between  intensity  and  time  spent  with  amount  of  learning  on  a 
topic,  it  is  the  aim  of  effective  instruction  to  keep  the  student's  mind 
engaged  in  the  subject  matter.  Daydreaming  cannot  count  as  study  time. 
Appropriate  teaching  techniques  and  appropiate  practice  materials  can  elicit 
the  high  degree  of  involvement  necessary  in  students  to  hasten  them  along 
paths  of  learning.  Breaks  in  the  class  period  can  be  productive  in  PAR  only 
if  the  students  talk  about  that  which  they  have  been  doing. 

4.  Re i nf orcemen t  ( RNF ) .  Reinforcers  strengthen  the  behavior  that 
precedes  them.  They  should  be  given  after  appropriate  behavior  and  withheld 
after  inappropriate  behavior.  Reinforcers  are  idiosyncratic  to  students; 
however,  there  are  some  reinforcers  that  are  generally  applicable.  These 
include  praise,  recognition,  special  privilege,  and,  indirectly,  feelings  of 
achievement. 

5.  Feedback  (FBK).  Providing  students  with  information  about 
performance  serves  not  only  to  reinforce  successful  behavior,  but  it  also 
provides  guidance  on  what  to  study  and  how  much  effort  to  expend  in  meeting 
course  goals.  Tests,  quizzes,  self  tests,  critiques,  oral  and  comprehensive 
examinations  all  provide  the  student  information  as  to  how  they  are  doing 
with  relation  to  enabling  and  terminal  learning  objectives.  Such  activities 
are  an  important  part  in  the  design  of  instructional  materials. 

6.  Correctives  (COR).  After  feedback  shows  the  student  that  there  is 
a  difference  between  the  demonstrated  and  the  required  performance  or 
practice,  adaptive  instruction  calls  for  a  prescription  to  get  the  student 
back  on  track.  Correctives  are  the  prescribed  alternate  forms  of  presenting 
that  which  the  student  is  to  learn.  The  summary,  narrative,  and  programmed 
instruction  mentioned  in  NAVEDTRA  110A  could  be  used  as  correctives  in 
certain  situations.  Correctives  are  the  learning  activities  that  adaptive 
instruction  uses  to  ensure  that  all  students  have  repeated  opportunities  to 
learn. 
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PLAN  OF  THE  STUDY 

The  study  was  conducted  in  the  following  sequence  of  activities: 

•  conducted  a  review  of  the  literature  concerninq.il  (presented  in 
appendix  E) 

•  distributed  a  Course  Description  survey  to  a  population  of  201 
courses  categorized  as  individualized  instruction 

•  established  categories  for  assessing  .  the  variations  in 
instructional  practices 

•  determined  a  sample  of  courses  for  site  visits  and  detailed 
analysis  of  instructional  practices  and  materials  used 

•  site  visited  37  courses  and  administered  a  structured  interview 
and  quality  of  instruction  questionnaire  to  a  sample  of  students, 
instructors,  and  supervisors 

•  categorized  courses  as  conventional,  mixed,  or  self-paced 
instruction  and  examined  the  degree  to  which  elements  of  quality 
instruction  in  each  category  were  present. 

ORGANIZATION  OF  THE  REPORT 

In  addition  to. this  introduction  the  report  contains  four  sections  and 
six  appendices.  Section  II  provides  a  detailed  description  of  the  approach 
taken  in  the  study,  and  also  describes  the  sample  and  instruments  used  to 

gather  data.  Section  III  presents  the  results  of  the  analyses  of  data. 
Section  IV  is  a  discussion  of  the  results.  Section  V  provides  conclusions 
and  recommendations.  Appendix  A  contains  the  Course  Description  Form  with  a 
tabulation  of  responses  to  each  item.  Appendix  B  presents  the  Structured 
Interview  Form  with  a  summary  of  course  data  organized  by  three  categories 
of  II.  Appendix  C  contains  the  Quality  of  Instruction  Questionnaire  and  the 
scaled  values  for  each  item.  Appendix  D  presents  the  mean  values  for  each 
item  of  the  Quality  of  Instruction  Questionnaire  as  a  function  of  type  of 
II.  Appendix  E  gives  a  review  of  literature  pertinent  to  the  rationale  for 
using  the  Quality  of  Instruction  variables  in  the  study.  Appendix  F 
synthesizes  the  literature  and  study  findings  with  a  model'  of  ideal 
individualized  instruction. 
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SECTION  II 
TECHNICAL  APPROACH 

This  section  describes  the  sample,  the  development  of  the  Course 
Description  Form  used  to  obtain  information  describing  the  various  types  of 
II  courses,  the  procedure  for  deciding  which  courses  to  visit  for  more 
detailed  information,  the  development  of  a  quality  of  instruction 
questionnaire  used  to  assess  instructional  practices  within  the  variations 
of  II,  an  analysis  of  student  time  taken  to  master  instruction,  a  structured 
interview  procedure,  and  the  major  data  analysis  procedures.  Throughout 
this  section  there  are  descriptions  of  salient  characteristics  of  courses 
falling  in  each  of  three  categories,  or  variations  of  individualized 
instruction  ( VAR II):  conventional,  mixed,  and  self-paced. 

DESCRIPTION  OF  THE  SAMPLE 

The  study  began  with  a  survey  of  courses  in  the  FY  1980  Master  Course 
Reference  File  of  the  Navy  Integrated  Training  and  Resources  Administration 
System  (NITRAS)  which  were  listed  as  self-paced,  computer-managed  (CMI),  or 
having  a  combination  of  methods  of  individualizing  instruction.  This  search 
yielded  199  Course  Data  Processing  (CDP)  numbers,  of  which  69  were 
instructor-managed  instruction,  11  were  computer-managed  instruction,  and 
119  were  ,  a  combination  of  these  two  methods.  These  methods  were  listed  in 
NITRAS  as  "P,"  "C,"  or  "B,"  respectively. 

To  select  the  sample  of  courses  for  more  detailed  analysis,  a  Course 
Description  Form  (CDF)  was  designed  to  elicit  information  concerning 

instructional  practices  being  used  in  courses  identified  in  NITRAS  as 

containing  some  degree  of  II.  The  questions  generally  dealt  with  time, 
administration,  and  philosophies: 

1.  The  use  of  instructor,  student,  study,  and  awaiting  instruction 

time:  Specific  questions  deal  with  use,  recording,  and 

perceptions  of  these  various  types  of  time. 

2.  Course  administration  and  organization:  This  category  of  questions 

includes  such  concepts  as  criterion  for  mastery,  use  of  feedback, 

types  of  reinforcement  employed,  media  and  instructional  material 
used,  numbers  and  type  of  personnel  assigned,  and  relative 

proportions  of  instructional  type  (lecture,  lab,  self-pacing). 

3.  Training  philosophies:  This  group  of  questions  assessed  beliefs 

and  attitudes  about  the  basic  ability  of  students  to  learn,  the 

level  of  achievement  to  be  expected  of  a  cohort  of  students,  and 
the  control  over  pacing  of  students. 

See  appendix  A  for  specific  questions  dealing  with  each  of  these  areas. 

CDFs  were  mailed  to  83  of  the  199  potential  target  courses.  Only  one 
CDF  was  mailed  to  courses  having  identical  course  identification  numbers  and 
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titles  but  varying  CDPs.  at  the  same  location.  An  example  is  the  Basic 
Electricity  and  Electronics  (BE&E)  course,  which  was  comprised  of  24  CDPs  in 
Orlando,  30  at  Great  Lakes,  27  at  San  Diego,  and  9  at  Memphis.  Each 
location  received  only  one  CDF.  Table  1  summarizes  the  number  of  courses  at 
each  location  and  the  number  of  CDFs  sent  and  used  in  this  study.  Included 
in  table  1  are  two  courses  that  later  became  available  from  the  Marine  Corps 
Communication-Electronics  School  (MCCES).  They  were:  a  computer-assisted 
and  an  instructor-managed  course  from  MCCES,  Twentynine  Palms,  California. 
Both  were  versions  of  the  Communications  Center  Operator’s  Course  (CCOC) 
taught  by  differing  methods.  Of  the  83  CDFs  sent,  78  were  returned  or 
accounted  for--a  94  percent  return  rate.  Based  on  these  returns,  64  courses 
were  suitable  for  analysis. 

TABLE  1.  POPULATION  OF  NITRAS  COURSES  BY  LOCATION 


Location 

Number  of  P, 
C,  or  B 
Courses 

Number  of  Course 
Description 
Forms  Sent 

Number  of  CDFs 
Included 
in  Study 

*  SSC  Orlando,  FL 

27 

2 

2 

*  NTTC  Corry  Station,  FL 

14 

13 

12 

*  NATTC  Memphis,  TN 

35 

6 

6 

SSC  Great  Lakes,  IL 

48 

16 

11 

*  SSC  San  Diego,  CA 

34 

8 

5 

FTC  San  Diego,  CA 

7 

7 

1 

*  NTTC  Meridian,  MS 

5 

5 

5 

*  FTC  Charleston,  SC 

3 

3 

2 

STC  Charleston,  SC 

2 

2 

2 

NTTC  Treasure  Island,  CA 

2 

2 

2 

*  FTC  Philadelphia,  PA 

4 

4 

4 

NSS  Groton,  CT 

3 

3 

3 

NATTC  Lakehurst,  NJ 

9 

6 

6 

FTC  Norfolk,  VA 

2 

2 

o  ! 

NAMT  Millington,  TN 

1 

1 

l  , 

STC  PAC  Pearl  Harbor,  HI 

3 

3 

o 

**MCCES  29  Palms,  CA 

2 

2 

2 

Total 

4-  1 _ j* _  .  ■  •  .  .  i 

201 

85 

64 

*  Locations  visited. 

**Sampled  course  not  in  NITRAS. 
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Table  2  lists  the  64  courses  from  which  usable  CDFs  were  obtained.  The 
table  gives  a  sequence  number  depicting  the  order  the  completed 
questionnaires  were  returned,  the  CDP  numbers,  course  short  title,  location, 
the  estimated  percent  of  the  course  that  was  "self-paced"  (Question  No.  31 
on  the  CDF),  and  the-  number  of  students  in  annual  planned  input.  In 
addition,  the  courses  receiving  a  site  visit  are  indicated  by  an  asterisk. 

The  CDF  was  sent  to  the  person  "most  knowledgeable"  of  the  selected 
course.  A  cluster  analysis  of  the  responses  to  the  original  62  Navy  CDFs 
did  not  reveal  clear  patterns  associated  with  variations  in  II;  however, 
this  information  was  used  to  select  the  widest  variety  of  instructional 
practices  in  courses  at  a  given  training  location.  CDF  question  No.  31, 
"What  percent  of  this  course  is  'self-paced'?"  had  the  highest  relationship 
with  variations  in  II  (VARII).  Thus,  this  question  became  the  basis  for  the 
categories  of  II  used  in  this  study.  The  percent  of  self-pacing  in  each  of 
the  64  courses  from  which  CDFs  were  used  is  listed  in  table  2. 

DESCRIPTION  OF  THE  CATEGORIES  OF  II 

Three  VARII  were  identified.  There  were  7,  10,  and  20  courses, 
respectively,  in  the  three  categories.  Those  courses  reporting  0-4  percent 
self-paced  on  CDF  question  31  were  classified  as  conventional  instruction 
(Cl).  Those  courses  5  to  90  percent  self-paced  were  classified  mixed  (MIX). 
Courses  with  91  percent  or  greater  were  self-paced  (SP).  The  analyses 
described  here  and  the  results  reported  in  section  III  are  based  on  these 
categories. 

Additional  data  from  the  CDF  were  analyzed  to  provide  a  more  detailed 
description  of  courses  in  the  three  VARII.  Tabulations  were  made  for  the 
CDF  questions  eliciting  frequency  data,  and  univariate  analyses  of  variance 
were  utilized  for  the  interval-  and  ratio-scaled  data.  The  following 
describes  some  additional  characteristics  of  the  three  categories  of 
courses.  The  most  individualized  courses  appear  to  be  more  likely  to  let 
the  student  determine  when  breaks  were  to  occur  during  the  class  day.  In 
addition,  these  courses  show  more  variability  in  the  time  taken  to  finish 
the  course.  The  three  VARII  differed  in  the  hours  in  a  typical  class  day, 
the  learning  minutes  in  a  typical  class  hour,  the  criterion  for  mastery  on  a 
lesson  or  module,  the  amount  of  lecturing,  the  amount  of  self-study,  and  the 
use  of  study  materials.  Tables  3,  4,  and  5  provide  summary  data  for  these 
findings. 
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TABLE  2.  COURSE  POPULATION 


Seq  CDP(s) 

No. 

Short 

Title 

Loca¬ 

tion 

%SP 

(Q31) 

Planned 

Input 

*  1.  3665-3698  (3) 

TTA 

Orlando 

100 

64 

*  2.  604H-6550  (24) 

BE/E 

Orlando 

100 

3708 

*  3.  303N 

CTO  TACSOM 

Corry 

0 

140 

*  4.  3197 

CTT  EL I NT  OP 

Corry 

75 

100 

*  5.  6458 

CTT  WBS  OP 

Corry 

90 

100 

*  6.  417M 

NSG  JOOC 

Corry 

0 

70 

7.  6059 

SK  A 

Meridian 

100 

1079 

*8.  6522 

AK  A 

Meridi an 

90 

722 

*  9.  6102 

PN  A 

Meridian 

100 

1263 

10.  6057 

YN  A 

Meridian 

100 

1844 

*11.  6061 

DK  A 

Meridian 

100 

350 

*12.  6501 

AD  A1 

Memph i s 

60 

1832 

13.  6161 

CTM  A 

Corry 

5 

405 

*14.  6320 

CTT  SPE 

Corry 

5 

700 

15.  6302 

CTT  A  Prep 

Corry 

0 

1325 

16.  4376 

CTT  SNMC/O 

Corry 

100 

50 

17.  6319 

CTT/ICR/Flex 

Corry 

0 

100 

*18.  6380 

RM  A  Sea 

San  Diego 

0 

1582 

*19.  6381 

RM  A  Shore 

San  Diego 

0 

1545 

20.  9332 

PCO/PXO  Rev 

San  Diego 

30 

15 

21.  281V 

Corosion  Cont 

Mill ington 

100 

*** 

*22.  403V 

LC  Instr 

Great  Lakes 

100 

350 

*23.  5382 

MSE  CODE  OP 

San  Diego 

100 

400 

*24.  601R-6551  (30) 

BE/E 

Great  Lakes 

100 

4761 

*25.  6144 

RM  A  Basic 

San  Diego 

100 

3677 

*26.  5152 

BAS  DIG  Fund 

Charleston 

100 

295 

*27.  5202 

GPETE  Bas  Op 

Charleston 

100 

375 

*28.  6269-6549  (27) 

BE/E 

San  Diego 

100 

5654 

29.  6119 

HT  A-l 

Treasure  IS 

0 

1124 

30.  2589 

FUEL  PROBE 

Treasure  IS 

0 

85 

31.  6301 

CTR  A 

Corry 

0 

615 

32.  6020 

CTA  A 

Corry 

0 

240 

*33.  8511 

GAS  FREE  ENG 

Philadelphia 

0 

75 

34.  5340 

DC  REP  PTY  LDR 

Philadelphia 

33 

83 

*35.  6339 

HT  A-2 

Phi ladelphi a 

99 

1086 

*36.  3218 

DC  ASST 

Philadelphia 

80 

198 

37.  2859 

BECTEC 

Great  Lakes 

0 

304 

38.  6492 

MM CL A1 200 

Great  Lakes 

85 

2250 

;  39.  6493 

MMCLA  600 

Great  Lakes 

90 

2737 

*40.  6486 

BT  CL  A  1200 

Great  Lakes 

90 

2250 

*41.  6488 

BT  ADV  OPER 

Great  Lakes 

0 

600 

*42.  6261-8562  (4) 

PE 

Great  Lakes 

100 

10248 

*43.  6487 

EN  CL  A 

Great  Lakes 

65 

1861 

44.  6280-6284  (4) 

AV  FUN 

Lakehurst 

85 

1605 
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TABLE  2.  COURSE  POPULATION  (continued) 


Seq  CDP(s) 

No. 

Short 

Title 

Loca¬ 

tion 

%SP 

(Q31) 

Planned 

Input 

45.  6519 

PR  Bas 

Lakehurst 

0 

579 

46.  4509 

PR  Adv 

Lakehurst 

0 

102 

47.  7764 

NP/I/ 

Lakehurst 

0 

139 

48.  7765 

NP/1 1/ 

Lakehurst 

0 

2 

49.  7766 

NP/I 11/ 

Lakehurst 

0 

2 

*50.  602D 

EW  A 

Corry 

87 

912 

51.  2694-6046  (3) 

IM  A 

Great  Lakes 

99 

79 

*52.  8981 

TMA/SSPP 

Charleston 

50 

240 

*53.  540J 

CIAC 

Charleston 

0 

460 

*54.  604J-6243  (6) 

AV  A 

Memphis 

100 

4600 

*55.  601B-6237  (9) 

BE/E 

Memphis 

100 

7855 

*56.  6210-6229  (17) 

AFUN 

Memph i s 

100 

16860 

57.  6521 

TD  A1 

Memph i s 

100 

361 

*58.  402P 

LC  INSTR 

Memphis 

100 

195 

59.  501X 

GPETE  Bas  Op 

Groton 

100 

392 

60.  022B 

DIESEL  Op 

Groton 

0 

218 

61.  501C 

OX  ANAL  C  CMB 

MA  Groton 

100 

16 

*62.  6047 

OM  A 

Great  Lakes 

100 

57 

*63.  USMC 

CCOC  (CAI) 

29  Palms 

100 

214* ** *** 

*64.  USMC  ’ 

CCOC  ( IMI) 

29  Palms 

100 

434** 

*  Course  visited. 

**  FY  82  throughput. 

***Course  ended  4  Sept  80. 


13 


Technical  Report  147 


TABLE  3.  MEAN  CLASS  TIME  AND  CRITERION  TO  MASTERY  FOR 
THREE  VARII  (CDF  NOS.  3,  4,  14) 


ACTIVITY 

Cl 

MIX 

SP 

Class  Day/Hours 

7.71 

7.40 

6.85 

Class  Hour/Min. 

48.14 

50.80 

54.15 

Criterion  to 

Mastery  % 

73.00 

75.40 

93.05 

(The  three  VARII 

differed  significantly  on 

all  activities.) 

TABLE  4.  PERCENT  ACADEMIC  DAY  SPENT  IN 
CENTER  ACTIVITIES  (CDF  NO. 

THREE  VARII  BY  SEVEN  LEARNING 
23)* 

ACTIVITY 

Cl 

MIX 

SP 

Lecture 

54 

11 

2 

Discussion 

8 

4 

3 

Demonstration 

4 

5 

3 

Self-Study 

7 

53 

53 

Tutori ng 

0 

2 

3 

Film 

3 

3 

1 

Laboratories 

24 

22 

32 

Other 

0 

1 

3- 

! 

*Does  not  add  to  100  due  to  rounding. 

TABLE  5. 

PERCENT  CLASSROOM  SELF-STUDY  TIME 
INDIVIDUALIZATION  BY  SIX  TYPES  OF 
(CDF  NO.  25) 

IN  THREE  LEVELS  OF 

STUDY  MATERIALS 

ACTIVITY 

Cl 

MIX 

SP 

Student  guides  . 

16 

9 

17 

Sum,  Nar,  P.I. 

0 

56 

51 

Other  P.I. 

0 

0 

4 

Handouts 

48 

15 

13 

Equipment  Manuals 

14 

1 

7 

Other 

16 

9 

5 
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An  important  difference  in  the  variations  of  II  is  in  the  ratio  of 
students  to  instructors  and  supervisors  (table  6).  The  SP  learning  centers 
have  over  five  times  as  many  students,  on  the  average,  as  do  the  Cl  classes, 
while  there  are  only  about  twice  as  many  instructors  present.  The 
student/instructor  ratio  (S/I)  for  SP  is  about  18  to  1,  while  in  the  Cl 
environment  it  is  about  8  to  1. 


TABLE  6.  MEAN  NUMBER  OF  PERSONNEL  IN  CLASSROOM  OR  LEARNING  CENTER 


VARII  . 

N 

SDNTS 

AIDES 

INST 

SUPERV 

S/I 

Cl 

7 

8.86 

0.14 

1.14 

1.86 

7.77 

MIX 

10 

19.50 

0.30 

2.00 

2.60 

9.75 

SP 

20 

45.30 

0.60 

2.50 

3.70 

18.12 

Total 

37 

31.43 

0.46 

2.11 

3.05 

14.90 

Of  the  64  courses  summarized  in  table  6,  16  (25  percent)  had  S/I  ratios 
of  six  or  less.  These  courses  break  down  to  nine  Cl  (56  percent),  three  MIX 
(19  percent),  and  four  SP  (25  percent). 

QUALITY  OF  INSTRUCTION  QUESTIONNAIRE 

A  quality  of  instruction  (QI)  questionnaire  was  developed  to  assess  the 
extent  to  which  prerequisites  (PRQ),  cues  (CUE),  participation  (PAR), 
reinforcement  (RNF),  feedback  (FBK),  and  correctives  (COR)  are  present  in 
various  types  of  individualized  instruction.  Items  were  constructed 
according  to  Bloom's  (1976)  theories  of  Learning  for  Mastery  and  were 
adapted  to  military  training  situations.  Each  item  was  examined  for  content 
and  structure  by  TAEG  staff  members.  The  interim  QI  was  field  tested  with 
students  from  the  Orlando  BE/E  course.  In  addition,  instructors  and 
supervisors  were  asked  to  review  the  questions.  The  final  version  of  the  QI 
used  in  the  study  was  based  on  suggestions  from  these  students,  instructors, 
and  supervisors.  The  resulting  questionnaire  consisted  of  50  items  that 
could  be  administered  in  approximately  20  minutes.  Appendix  C  presents  this 
questionnaire  with  the  obtained  scale  value  for  each  of  the  variations  of  II 
on  each  item. 

The  QI.  questionnaire  was  administered  to  1,090  students,  170 
instructors,  and  54  supervisors  involved  with  37  courses  at  9  training 
sites.  The  combined  groups  yielded  an  alpha  reliability  coefficient  of  .86 
on  the  instrument.  The  scoring  templates  were  scaled  1  to  9  for  data 
analysis  with  the  polarity  reversed  on  appropriate  scales  so  that  results 
could  be  consistently  interpreted.  The  questions  were  grouped  according  to 
the  six  QI  categories  (Bloom,  1976)  to  facilitate  the  examination  of  the 
impact  of  these  variables: 
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•  Prerequisites  (Q1-Q5,  07,  Q25) 

•  Cues  (Q8-Q14) 

•  Participation  (Q16,  Q18-Q20,  Q22,  Q23,  Q26-Q30) 

•  Reinforcement  (Q3.1-Q38,  Q17,  Q21) 

•  Feedback  (Q40-Q44) 

•  Correctives  (Q39,  Q45-Q47,  Q49,  Q50). 

Later  analyses  of  questions  6,  15,  24,  and  48  found  them  not  as  related 
to  the  six  QI  variables  as  anticipated,  although  their  results  are  of  use  to 
the  study. 

Table  7  shows  the  distribution  of  QI  questionnaires  to  the  courses  in 
each  category  of  II  and  the  yearly  planned  student  input  for  each  of  these 
categories.  Data  obtained  from  these  questionnaires  were  submitted  to 
detailed  analyses  of  variance  of  VARII  and  type  of  respondent  by  each  of  the 
50  questions  and  questions  grouped  according  to  the  six  elements  of  QI. 


TABLE  7.  NUMBER  OF  COURSES,  QUESTIONNAIRES  ADMINISTERED,  AND 


PLANNED 

INPUT  FOR  COURSES 

VISITED 

VARII 

(N  COURSES) 

QI 

(N  RESPONDENTS) 

PLANNED 

INPUT 

%  PLANNED 
INPUT 

Cl 

7 

146 

4,472 

6 

MIX 

10 

210 

8,915 

12 

SP 

20 

958 

62,489 

.  82 

TOTAL 

37 

1,314 

75,876 

100 

TIME  TO  MASTERY  ANALYSIS 

The  literature  of  Learning  for  Mastery  predicts  that  the  time  required 
for  learning  in  a  diverse  group  of  students  will  become  more  homogeneous  the 
longer  they  participate  in  efficient  instruction  (Bloom,  1976;  Anderson, 
1976).  This  is  contrary  to  the  traditional  notion  that  student  achievement 
becomes  more  heterogeneous  the  longer  they  stay  in  school.  One  aspect  of 
the  present  study  was  to  search  for  an  index  of  such  homogeneity  in  time 
required  for  learning  as  a  measure  of  instructional  efficiency.  The 
coefficient  of  variability  (v  =  s/m)  was  chosen  as  the  index;  where,  v  = 
variability,  s  =  standard  deviation  of  student  time  to  mastery  on  a  given 
module,  and  m  =  mean  student  time  to  mastery  on  the  module.  If  students 
requiring  longer  time  for  learning  begin  an  individualized  course  of 
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study  and  become  more  and  more  like  the  faster  learners,  then  the  standard 
deviation  should  decrease  for  a  given  cohort  as  they  progress  from  module  to 
module.  Unfortunately,  these  modules  are  not  of  the  same  length  or 
difficulty,  so  this  makes  it  possible  for  the  variability  of  the  longer 
modules  to  increase.  The  coefficient  of  variability  tends  to  correct  such 
effects.  This  procedure  was  applied  to  a  selected  sample  of  courses  for 
which  there  were  time  to  mastery  raw  data  available. 

STRUCTURED  INTERVIEW 

In  addition  to  the  quality  of  instruction  questionnaire,  a  structured 
interview  was  developed  to  specifically  assess  the  degree  to  which  Bloom's 
elements  of  quality  instruction  were  present  in  the  instructional  materials. 
Consequently,  the  structured  interview  had  six  major  sections,  each  dealing 
with  an  element  of  01  (see  appendix  B) .  The  structured  interviews  were 
administered  to  the  person  deemed  "most  knowledgeable"  of  the  course  by  one 
of  the  two  principal  investigators.  Analyses  of  the  responses  are  based  on 
a  total  of  37  interviews,  one  for  each  of  the  courses  sampled. 

In  conjunction  with  these  interviews,  instructional  materials  for  each 
of  the  courses  were  physically  examined  and  assessed  for  the  degree  to  which 
elements  of  QI  were  present.  Finally,  samples  of  reading  materials  were 
subjected  to  a  Computer  Readability  Editing  System  (CRES)  analysis  (Kincaid, 
Aagard,  and  O'Hara,  1980).  Data  from  the  interviews  were  analyzed  by  cross 
tabulating  the  VARII  with  each  question  and  the  six  01  groups  of  questions. 

EVALUATION  OF  NAVEDTRA  110  (SERIES)  GUIDELINES 

A  separate  analysis  was  conducted  of  NAVEDTRA  110  (series)  guidance 
with  respect  to  Quality  of  Instruction  elements.  For  the  purposes  of  this 
analysis,  each  question  of  the  structured  interview  was  considered  to  be  an 
aspect  of  the  01  elements  for  which  guidance  could  be  prepared.  Aspects  of 
each  element  of  QI  were  assessed  for  their  correspondence  to  sections  of  the 
draft  NAVEDiRA  HOB,  the  proposed  revision  to  NAVEDTRA  110A,  which  is  the 
current  instruction  guiding  the  development  of  instructional  materials  in 
the  NAVEDTRAC0M.  Based  on  the  discrepancies  observed,  recommendations  for 
modifications  to  the  NAVEDTRA  110  (series)  instruction  and  for  its  use  were 
developed. 
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SECTION  III 
RESULTS 

This  section  summarizes  the  results  of  the  QI  questionnaire  analyses, 
time  and  costs  analyses,  the  examination  of  course  materials  with  the 
structured  interview,  and  on-site  observations. 

QUALITY  OF  INSTRUCTION 

The  mean  score  for  respondents  in  each  VARII  on  45  of  the  50  items  on 
the  QI  questionnaire  was  above  average.  This  indicates  generally  favorable 
opinions  toward  almost  every  aspect  of  instruction  measured  by  this 
questionnaire.  The  five  unfavorable  exceptions  (Q7,  20,  23,  26,  45),  cut 
across  categories  of  QI  and  deal  with  highly  specific  aspects  of  instruction 
as  opposed  to  generalized  trends;  the  most  notable  of  these  is  the  relative 
ease  with  which  students  can  daydream  in  self-paced  instruction  (Q23).  The 
most  favorable  responses  showed  that  instructors  were  highly  regarded  as 
subject  matter  experts  (Qll)  and  showed  agreement  in  the  usefulness  of 
practice  activities  for  students  (Q29).  The  reader  is  referred  to  appendix 
C  for  the  mean  scaled  responses  on  each  item  for  the  three  VARII. 

The  QI  items  were  combined  to  obtain  information  concerning  the  six 
elements  of  quality  instruction,  as  mentioned  previously  in  this  section. 
These  six  QI  (PRQ,  CUE,  PAR,  RNF,  F8K,  COR)  became  the  dependent  measures 
for  3X3  ANOVAs.  The  independent  variaoles  were  3  VARII  (Cl,  MIX,  SP)  X  3 
respondent  categories  (student,  instructor,  supervisor). 

Figure  1  shows  QI  scores  as  a  function  of  VARII.  Analysis  of  the  data 
indicates  Cl  to  be  judged  by  respondents  as  providing  the  highest  degrees  of 
CUE,  PAR,  and  RNF  relative  to  MIX  and  SP.  MIX  courses  were  rated  higher 
than  Cl  and  SP  in  PRQ  and  COR.  SP  was  estimated  to  be  superior  to  Cl  and 
MIX  only  in  feedback.  A  more  detailed  analysis  of  this  main  effect  is 
presented  in  the  following  paragraphs.  Appendix  C  gives  a  breakdown  by 
questionnaire  item  and  appendix  0  gives  the  mean  values  and  ANOVA  results 
for  the  VARII  main  effect, 

PREREQUISITES.  The  mean  response  to  the  seven  questions  comprising  PRQ  on 
the  QI,  shown  in  figure  1,  differed  significantly  among  the  three  VARII, 
with  the  MIX  and  SP  groups  rated  highest.  This  was  probably  due  to  the 
respondents  in  these  courses  finding  the  reading  levels  of  student  materials 
easier  (Q2),  perceiving  that  more  of  the  students  are  high  in  background  for 
success  in  the  course  (Q5),  and  students  being  far  more  likely  to  progress 
through  the  course  at  a  rate  of  speed  commensurate  with  their  prerequisites 
for  the  task  (Q25).  This  is  in  spite  of  the  fact  that  the  Cl  students  have 
better  attitudes  about  school  learning  going  into  their  courses  (Q4j. 

CUES.  The  seven  CUE  questions  combined  to  show  the  Cl  courses  highest,  as 
shown  in  figure  1.  Here,  there  was  the  perception  among  respondents  that 
the  learning  objectives  were  more  specific  (Q9),  there  was  far  more 
instructor  assistance  in  holding  the  students'  attention  to  the 
instructional  materials  (Q10),  and  there  was  a  greater  proportion  of  the  Cl 
instructor's  day  being  spent  in  giving  cues  to  students  (Q14). 
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PARTICIPATION.  The  11  PAR  questions  taken  together  showed  the  Cl  group 
highest  overall  in  this  attribute  of  quality  instruction  (figure  1).  Here, 
the  Cl  respondents  perceived  students  to  spend  significantly  more  time 
beyond  the  scheduled  academic  work  day  on  their  courses  (Q20),  perceived  the 
teaching  methods  in  their  courses  to  be  more  appropriate  (022),  felt  it  was 
more  difficult  to  daydream  in  class  (023),  detected  a  greater  degree  of 
instructor  participation  in  determining  the  rate  of  student  progress  (026), 
saw  more  time  spent  practicing  what  students  learn  (Q27),  and  generally 
found  more  realistic  practice  activities  (Q28  to  Q30) .  The  SP  courses  were 
most  skillful  in  cutting  the  time  between  presentation  and  practice  (Q16) 
and  cutting  the  time  awaiting  start  of  their  courses  (Q19). 

REINFORCEMENT.  The  third,  and  last  of  the  01  variables  on  which  the  Cl 
group  scored  highest,  are  the  10  RNF  questions  (figure  1).  In  the  Cl 
courses,  the  instructors  were  perceived  as  more  enthusiastic  (Q31),  and  more 
likely  to  believe  that  all  students  can  and  will-  learn  (Q33,  Q34) . 
Graduates  of  the  courses  were  mere  likely  to  see  the  importance  of  the 
course  for  their  military  career  (Q37),  and  more  likely  to  recommend  the  Cl 
courses  to  others  (Q38).  The  lesson  materials  were  more  likely  to  explain 
the  importance  of  learning  their  contents  ( Q17 ) ,  and  were  perceived  to  hold 
interest  longer  (Q21). 

FEEDBACK.  The  SP  and  MIX  courses  scored  higher  than  Cl  on  the  five  combined 
FBK  questions  (figure  1).  These  courses  were  perceived  as  allowing  more 
tests  and  retakes  of  examinations  (Q41  and  Q42),  even  though  there  seems  to 
be  less  instructor  involvement  following  a  test  (Q44). 

CORRECTIVES.  The  combined  responses  to  the  six  COR  questions  shown  in 
figure  1,  find  the  SP  group  lowest.  This  appears  to  be  an  anomaly  since  the 
theories  providing  the  foundations  for  Mastery  Learning  emphasize  the 
importance  of  corrective  activity.  The  anomaly  is  probably  due  to  the  SP 
respondents'  feeling  that  the  time  allowed  for  relearning  after  a  failed 
examination  is  too  short  (Q45),  that  there  is  less  availability  of 
instructors  for  helping  students  (Q46),  and  the  perception  of  less 
availability  of  practice  and  restudy  materials  (Q49). 

Figure  2  shows  the  relationships  of  the  three  types  of  respondent 
(student,  instructor,  supervisor)  and  the  six  QI  variables  summed  across 
VARII.  There  were  significant  differences  among  respondents  in  four  of  the 
six  QI  categories.  The  supervisors  generally  perceived  the  greatest  degree 
of  PRQ,  RNF,  FBK,  and  COR  present  in  their  courses,  the  instructors  were 
highest  in  the  PAR- present,  and  there  were  no  significant  differences  among 
the  groups  on  the  CUE  and  RNF  questions.  Finally,  there  were  no  significant 
interactions  on  any  of  the  six  dependent  variables  undergoing  analyses  of 
variance. 
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7.5 


PRQ  CUE  PAR  RNF  FBK 


LEGEND: 

SOLID=CONVENTIONAL  INSTRUCTION 
SHADED  =  MIXED 

OUTLINED  =  SELF-PACED  INSTRUCTION 


Figure  1.  Three  VAR  IE  by  Six  QX 


COR 
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7.5 


LEGEND: 

SOLID = Students 
SHADED  =  Instructors 
OUTLINE = Supervisors 


Figure  2.  Profiles  of  1,083  Students,  164  Instructors, 
and  52  Supervisors  on  Six  QI  Variables 
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TIME  TO  MASTERY  ANALYSIS 

Time  to  Mastery  (TTM)  data  were  collected  from  three  sources:  (1)  all 
Phase  I  Basic  Electricity/Electronics  students  during  calendar  year  1981  (N= 
22704),  (2)  a  sample  from  Propulsion  Engineering  Basics  (N=62),  and  (3)  a 
sample  from  Opticalman  A  School  (N=19).  these  sources  represent  14,  15,  and 
22  modules,  respectively.  All  data  were  transformed  to  coefficients  of 
variability.  Figure  3  shows  a  plot  of  these  coefficients  across  the  number 
of  modules  representing  each  course.  A  fourth  source  was  taken  from  BE/E 
data  presented  in  Federico  and  Landis  (1979)  and  is  also  plotted  (FL)  in 
figure  3.  The  general  trend  of  each  of  the  lines  suggests  the  decreasing 
variability  of  TTM  that  Bloom  mentions,  but  the  rate  of  decrease  in 
coefficient  of  variability  is  unclear. 

COST  ANALYSIS 

The  results  of  the  foregoing  analyses  prompted  collection  of  cost  data. 
This  was  already  available  in  the  TAEG  Incremental  Costing  Model,  and  a 
breakdown  by  three  levels  of  VARII  is  shown  in  table  8  (Dickinson  and  Swope, 
1981).  The  three  costs  compared  here  are:  (1)  Total  cost  per  course  hour, 
(2)  Direct  costs  per  course  hour  (not  including  overhead),  and  (3)  Non¬ 
student  direct  costs  per  course  hour  (this  subtracts  student  salaries).  In 
all  three  comparisons  the  SP  costs  were  lower  than  the  MIX'  and  Cl  groups. 
This  is  coincident  with  the  changing  supervisor  ratio  reported  in  table  6; 
i.e.,  fewer  instructors  equal  lower  costs. 

TABLE  8.  PER  CAPITA  COST  DATA  FROM  1981  TAEG 
INCREMENTAL  COSTING  MODEL 


VARII 

Total  Cost/ 
Course  Hour 

Direct  Cost/ 
Course  Hour 

Non-Student 
Direct  Costs/ 
Course  Hour 

Cl 

$111.79 

$82.70 

$27.06 

MIX 

96.64 

71.33 

33.20 

SP 

64.48 

47.66 

9.67 

The  direct  costs  shown  on  table  8  are  not  as  heavily  affected  by 
student  throughput  as  are  total  costs,  although  there  may  be  some  influence 
of  throughput  on  these  costs.  A  comparison  of  the  average  throughput  of  the 
CDPs  in  the  present  study  shows  little  difference  among  the  Cl,  MIX,  and  SP 
categories.  This  may  be  due  to  some  of  the  larger  courses  in  the  study 
having  between  3  and  30  CDPs  per  course  location  (see  table  2).  The 
multiple  CDPs  per  large  course  tend  to  bring  the  throughput  per  Cl,  MIX,  and 
SP  category  toward  equality.  A  further  association  of  courses  with  these 
cost  findings  was  beyond  the  scope  of  the  present  tasking. 
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Figare  3.  CoefftckHrts  of  Variation  hi  Module  Time  to 
Mattery  to  Four  Seif-oaccd  Courses. 
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STRUCTURED  INTERVIEW 

Table  9  contains  a  summary  of  responses  to  the  structured  interview 
data  (appendix  B),  showing  the  extent  six  elements  of  QI  are  present  in 
instructional  materials  taken  from  the  37  courses  in  the  study.  The  data  in 
the  table  represent  percent  of  respondents  indicating  the  instructional 
element  was  present  in  their  course.  Two  types  of  values  appear  in  table  9. 
The  top  number  in  each  cell  is  the  mean  percent  of  the  pertinent 
questionnaire  items  answered  "yes,"  and  the  lower  numbers  show  the  range  in 
percent  of  "yes"  scores  for  course.  The  structured  interview  form  and  the 
percent  responses  by  VARII  category  are  shown  in  appendix  B. 


TABLE  9.  PERCENT  STRUCTURED  INTERVIEW  RESPONSES  OF 
THREE  VARII  BY  QI  VARIABLES 


CONDITIONS  OF  QUALITY  INSTRUCTION 


VARII 

PRQ 

CUE 

PAR 

RNF 

FBK 

COR 

Cl  Mean 
Range 

20 

0-50 

38 

14-62 

38 

0-88 

36 

0-100 

42 

25-67 

52 

33-100 

MIXED  Mean 
Range 

30 

0-50 

51 

24-86 

71 

13-100 

50 

0-100 

58 

42-83 

70 

0-100 

SP  Mean 
Range 

48 

13-75 

47 

19-86 

63 

0-100 

45 

0-100 

60 

33-92 

■  63 

0-100 

PREREQUISITES.  Observations  made  while  collecting  these  data  illustrate  how 
Navy  schools  attempt  to  deal  with  the  prerequisite  issue  of  assessing 
skills.  For  instance,  pretests  are  being  used  in  50  percent  of  the  visited 
self-paced  courses  to  determine  if  entering  students  have  the  required  math 
or  typing  skills.  Pretesting  is  a  useful  way  to  identify  those  who  require 
special  training  before  entering  the  main  part  of  a  course.  Some  courses 
allow  students  to  "test  out"  of  sections  of  the  course  without  further 
study,  if  students  can  convince  instructors  they  have  previously  mastered 
the  skills  being  taught.  However,  students  rarely  take  advantage  of  this 
offer. 

It  was  learned  that  as  much  as  50  percent  of  the  content  of  lessons  is 
isolated  information  not  applied  in  follow-on  lessons.  Comments  from  course 
specialists  frequently  included  statements  such  as  "...the  lessons  could  be 
taught  in  any  order,"  or  "...the  content  of  a  lesson  is  needed  in  a  follow- 
on  school,  but  would  not  be  used  again  in  the  current  course."  This 
suggests  that  many  courses  are  structured  so  as  to  exclude  distributed 
practice  in  the  application  of  newly  acquired  skills,  and  could  account  for 
low  retention  and  transfer  of  learned  skills  to  follow-on  courses  and  to  the 
job. 
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An  inspection  of  materials  revealed  that  self-paced  materials  appear  to 
be  better-written  than  materials  for  conventional  instruction.  None  of  the 
conventional  materials  passed  a  comprehensibility  check  while  63  percent  of 
the  self-paced  materials  passed  this  check.  Long  sentences  and  high  reading 
grade  levels  were  the  major  types  of  problems  found. 

CUES.  The  data  in  table  9  suggest  that  many  of  the  opportunities  to  present 
cues  in  instructional  materials  are  not  being  used.  This  is  true  for 
conventional  and  mixed  as  well  as  self-paced  instruction  (appendix  B) .  In 
the  materials  used,  opportunities  to  provide  various  types  of  cues  were  used 
only  38  percent  of  the  time.  The  use  rate  varied  from  14  to  62  percent  for 
individual  courses.  While  the  skilled  instructor  could  be  compensating  for 
these  deficiencies,  the  instructional  materials  did  not  support  these 
functions.  However,  within  self-paced  instruction  which  is  almost  entirely 
dependent  upon  the  instructional  materials  to  present  new  information,  an 
average  use  of  47  percent  of  the  possible  cue  functions  were  used.  For 
individual  courses  the  range  varied  from  19  to  86  percent  of  these  types  of 
cue  functions. 

Observations  made  while  collecting  these  data  provided  insight  into  the 
way  these  cue  functions  are  being  carried  out.  For  instance,  courses  vary 
widely  on  how  much  printed  information  a  student  is  given  to  provide  an 
overview  of  the  course.  While  43  percent  of  the  conventional  instruction 
courses  provided  students  with  course  outlines,  schedules,  student  profiles, 
lists  of  training  objectives  or  other  documents  that  summarize  what  will  be 
learned  in  the  course,  only  10  percent  of  the  self-paced  courses  provided 
these  types  of  materials  to  students. 

The  instructional  materials  in  these  courses  made  wide  use  of  both 
words  and  illustrations.  However,  there  were  important  instance_s  where 
there  is  a  mismatch  between  content  and  instructional  material.  The  most 
significant  of  these  mismatches  is  in  the  teaching  of  procedures.  Although 
performing  procedures  on  equipment  requires  locating  instruments  and 
controls,  and  repositioning  of  controls  based  on  visual  cues  (all  tasks 
requiring  extensive  processing  of  visual  information),  procedures  were 
generally  taught  by  having  students  read  written  steps.  Consequent  y, 
materials  to  teach  procedures  for  hands-on  equipment  operation  and 
maintenance  were  judged  to  be  the  least  effective  of  the  training  materials 
reviewed. 

In  general: 

•  locations  and  identification  cues  were  presented  by  words 

•  procedural  steps  were  frequently  unclear 

•  safety  practices  were  often  not  explicit 
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•  demonstrations  of  procedures  were  not  included  in  self-paced 
materials 

•  group-paced  demonstrations  were  difficult  to  follow. 

Other  types  of  cues  can  be  used  to  make  it  easier  for  students  to  learn 
and  recall  information.  Included  are  the  use  of  mnemonics  (memory  aids), 
dividing  information  into  easily  recallable  chunks,  the  use  of  memorable 
graphics,  and  highlighting  key  words  to  emphasize  those  words  that  when 
recalled  will  aid  students  in  remembering  related  material.  The  analysis  of 
instructional  materials  in  this  study  indicated  these  techniques  are 
infrequently  used  in  Navy  training  materials. 

PARTICIPATION.  Instructional  materials  were  also  examined  for  conformance 
to  eight  different  characteristics  of  good  practice  materials  (see  appendix 
B) .  Within  conventional  instruction  only  38  percent  of  the  recommended 
exercise  techniques  were  used  in  the  typical  courses.  The  mixed  courses 
averaged  using  71  percent  of  the  types  of  exercise  characteristics  while  the 
self-paced  courses  used  63  percent.  Individual  courses  ranged  from  0  to  88 
percent  for  conventional  instruction,  13  to  100  percent  for  the  mixed,  and  0 
to  100  percent  for  the  self-paced. 

Observations  of  good  practice  and  notes  on  how  to  improve  practice  were 
made  during  the  interviews.  Perhaps  the  most  useful  practice  technique 
observed  was  the  simulated  job  shop  used  at  the  conclusion  of  several  of  the 
courses.  These  shops  provide  opportunities  for  students  to  practice 
performing  the  job  they  will  be  assigned  on  arriving  at  their  new  duty 
station.  In  the  simulated  job  shop  students  have  the  time  and  resources  to 
perform  representative  tasks,  and  to  continue  to  do  this  until  certified  in 
this  performance. 

Distributed  practice  contributes  to  retention  and  the  ability  to  employ 
the  school -learned  skills  on  the  job.  In  some  courses  distributed  practice 
occurred  without  being  planned  or  supported  with  special  materials.  Courses 
sampled  rarely  contained  special  materials  for  distributed  practice. 

REINFORCEMENT.  The  instructional  materials  for  conventional  instruction 
used  36  percent  of  the  types  of  opportunities  to  employ  reinforcing 
functions,  while  the  materials  for  the  mixed  courses  used  50  percent  and  the 
self-paced  used  45  percent  of  these  opportunities.  Individual  courses  in 
all  three  types  of  instruction  varied  from  zero  to  100  percent  employment  of 
these  functions.  The  primary  observation  concerning  reinforcement  is  that 
little  attention  is  given  to  its  design  and  scheduling  in  Navy  training 
materials. 

FEEDBACK.  Scored  tests  are  an  important  form  of  feedback  to  students  on 
their  performance  in  a  course.  The  extent  of  feedback  provided  to  students 
in  a  course  can  be  estimated  by  the  types  and  the  frequency  of  tests  in  the 
course.  In  terms  of  the  present  study,  a  comprehensive  testing  program 
would  score  100  percent  on  the  use  of  feedback  opportunities.  The  actual 
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scores  were  42  percent  for  conventional  instruction,  58  percent  for  mixed, 
and  60  percent  for  self-paced  instruction.  Individual  course  scores  varied 
over  a  broad  range.  For  instance,  individual  self-paced  courses  scored  as 
low  as  33  percent  and  as  high  as  92  percent. 

CORRECTIVES.  Printed  directions  on  what  to  study  to  overcome  deficiencies 
discovered  through  testing  did  not  vary  greatly  across  the  three  types  of 
training  programs.  Conventional  instruction  used  52  percent  of  the  types  of 
opportunities,  while  mixed  and  self-paced  instruction  used  70  and  63  percent 
respectively. 

EVALUATION  OF  DRAFT  NAVEDTRA  110B  GUIDELINES 

In  general,  conformance  to  current  guidelines  in  draft  NAVEDTRA  110B 
would  correct  many  of  the  course  deficiencies  identified  in  this  study. 
However,  certain  elements  of  quality  instruction  call  for  guidance  not  dealt 
with  adequately  in  this  draft  instruction.  This  part  of  the  report 
documents  an  analysis  of  the  draft  NAVEDTRA  110B  directive  for  the  purpose 
of  identifying  areas  where  additional  guidance  is  needed. 

With  the  help  of  CNET  personnel  who  prepared  the  draft  110B,  the 
contents  of  the  instruction  were  compared  with  the  elements  of  quality 
instruction  on  the  structured  interview  for  evaluating  instructional 
materials.  In  this  manner  a  series  of  deficiencies  in  draft  110B  were 
identified. 

PREREQUISITES.  The  instruction  does  not  provide  the  necessary  requirement 
or  guidance  for  designers  to  properly  match  materials  to  student  prior 
learning.  It  does  not  require  that: 

•  pretests  be  used  to  determine  if  prerequisites  are  known 

•  prerequisites  for  a  lesson  be  taught  in  a  previous  lesson 

•  a  specified  reading  grade  level  or  comprehensibility  level  be 
maintained 

•  advanced  organizers  be  used  to  relate  previous  learning  to  a  new 
learning  task. 

CUES.  While  many  of  the  CUE-oriented  requirements  are  spelled  out  in  the 
proposed  instruction,  there  are  areas  that  need  to  be  expanded.  The 
instruction  does  not  require: 

•  course  overview  documents  be  given  so  that  students  understand  the 
content  and  flow  of  the  course  and  can  track  progress  through  the 
course 

•  appropriate  communication  channels  be  used;  i.e.,  verbal  informa¬ 
tion  with  words  and  visual  informat-ion  with  graphics 
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•  demonstrations  be  used  where  needed  and  standardized  through  the 
use  of  instructor  guides. 

PARTICIPATION.  Practice  of  newly  formed  skills  is  necessary  for  their 
retention  and  usefulness  on  the  job.  While  the  instruction  calls  for 
practice,  the  requirement  for  practice  and  its  support  needs  to  be  improved. 
The  proposed  instruction  does  not  require. 

•  students  to  distribute  practice  over  time  with  materials  designed 
to  support  distributed  practice. 

REINFORCEMENT.  It  is  necessary  to  reinforce  the  acts  of  studying  and 

acquiring  skilled  performance.  Although  reinforcement  is  essential  to 
learning,  it  is  not  addressed  in  the  proposed  instruction.  Specifically, 
the  proposed  instruction  does  not  require: 

•  instructional  materials  to  contain  statements  or  events  generally 
known  to  be  reinforcing  at  appropriate  points  in  the  instruction 

•  instructors  to  learn  skills  in  developing  reinforcement  menus  for 
individual  students  assigned  to  a  learning  center  with  clear 
instructions  on  how  to  shift  reinforcers,  withdraw  the  use  of 
external  reinforcers,  and  avoid  satiation  of  effective 
reinforcers. 

FEEDBACK.  In  addition  to  using  tests  to  determine  if  prerequisites  are 

present  in  students  beginning  an  instructional  module  or  course,  the  draft 
110B  does  not  require: 

•  curriculum  designers  to  consider  alternative  methods  of  providing 

formative  evaluation  to  learners,  such  as  oral  examination  and 

discussion  with  aides  or  advanced  students 

•  course  managers  to  maintain  reliability  and  content  validity  data 
on  formative  and  summative  examinations. 

CORRECTIVES.  The  draft  110B  does  not  describe  procedures  to  be  followed 
when  a  student  is  unsuccessful  in  either  formative  or  summative  examination. 
The  implication  is  that  such  students  go  back  and  restudy  the  same  materials 
using  the  same  methods  that  lead  to  failure  the  first  time.  While  the 

Narrative,  Summary,  and  Programmed  Instruction  hint  at  alternative  learning, 

they  do  not  provide  for  a  new  approach  to  teaching  the  subject,  with  new 
examples.  There  are  no  provisions  for  alternative  ways  of  attaining  the 
same  objectives. 
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SECTION  IV 
DISCUSSION 

This  section  discusses  the  findings  of  the  study.  First,  an  apparent 
contradiction  in  the  data  regarding  the  relative  effectiveness  of  ^ 

conventional  versus  self-paced  instruction  is  analyzed.  Next,  a  summary  of 
firdings  related  to  elements  of  quality  instruction  and  variation  of 
individualized  instruction  is  presented.  Cost  data  are  then  discussed  in 

terms  of  their  usefulness  in  assessing  the  efficiency  of  self-paced 
instruction. 

The  intent  of  this  study  was  to  examine  the  variations  in  instructional 
practices  in  the  courses  classified  in  NITRAS  as  individualized  instruction. 

The  data  in  section  III  show  that  there  are  indeed  differences  in  courses 
but  there  are  few  distinct  patterns  attributable  to  instructional  strategy; 
i.e.,  conventional  versus  self-paced.  The  differences  in  these  courses  are 
primarily  accounted  for  by  instructional  practices  which  can  be  for  the  most 

part  employed  irrespective  of  strategy;  that  is,  the  degree  to  which  they 

use  good  learning  principles.  As  defined  in  this  section,  II  has  three 
essential  ingredients;  (1)  releasing  of  time  constraints,  (2)  choosing 
instructional  media,  and  (3)  adjustment  to  skill  levels  and  learner 
characteristics.  The  NITRAS  categories  of  self-paced  instruction  (P), 
computer-managed  instruction  (C)  and  a  combination  of  the  two  (B),  depend  on 
self-pacing  to  determine  if  a  course  is  individualized.  This  can  lead  to 
errors  in  assessing  the  effectiveness  of  truly  individualized  instruction. 

The  data  in  section  III  also  show  an  apparent  discrepancy  in  findings 
resulting  from  the  use  of  the  quality  of  instruction  questionnaire  and  the 
structured  interview.  The  results  from  QI  generally  show  that  conventional 
instruction  is  superior  to  SP  in  the  provision  of  cues,  participation,  and 
reinforcement.  Conversely,  the  structured  interview  data  show  that  SP  is 
superior  to  Cl  in  all  six  elements  of  QI.  Part  of  this  discrepancy  may  be 
due  to  the  fact  that  these  time  measures  ask  for  different  information.  The 
QI  asks  for  perceptions  and  judgments  about  instructional  practices  while 
the  structured  interview  is  designed  to  assess  instructional  materials. 
Examination  of  table  9  indicates  that  on  a  relative  basis  SP  is  superior  to 
Cl.  On  an  absolute  basis,  however,  average  presence  of  QI  elements  reported 
for  material  is  far  from  what  could  be  considered  acceptable  in  any  of  the 
VAR 1 1 .  Cl  measures  ranged  from  20  to  52  percent,  the  MIX  range  was  30  to  71 
percent,  and  the  SP  range  was  45  to  63  percent.  One  interpretation  of  the 
superiority  of  Cl  over  SP  in  the  QI  questionnaire  is  that  in  a  conventional 
environment,  instructors  have  more  opportunity  and  appear  to  compensate  for 
deficiencies  in  instructional  materials.  This  interpretation  is 
strengthened  by  the  results  of  the  Johnson  and  Graham  (1982)  study  which 
showed  learning  center  instructors  spend  most  of  their  time  in  short  and 
routine  transactions  with  students  thus  precluding  their  opportunity  to 
provide  other  cues,  participation  and  reinforcement.  The  explanation  that 
instructors  will  compensate  for  curricular  deficiencies  is  further  supported 
by  the  data  in  table  5.  These  data  show  that  Cl  courses  in 
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the  present  study  had  lower  S/I  ratios,  therefore,  providing  the  chance  to 
compensate  for  deficiencies  in  materials. 

Additional  findings  of  interest  are  discussed  in  the  following 
paragraphs.  They  are  organized  by  the  six  elements  of  quality  instruction 
so  that  they  conform  to  the  approach  and  analyses  of  this  study. 

PREREQUISITES 

Questionnaire  respondents  in  all  variations  of  II  perceived  their 
courses  as  doing  an  adequate  job  adapting  to  individual  differences  in  prior 
learning,  intelligence,  and  attitudes  about  school.  Students  in  Cl  classes 
had  the  best  attitudes  about  school  learning  at  the  start  of  their  courses 
and  are  perceived  as  performing  at  a  higher  level  than  their  abilities  would 
allow.  Students  in  the  mixed  and  self-paced  classes  felt  their  study  habits 
were  most  improved.  Although  students,  instructors,  and  managers  gave  their 
courses  relatively  high  grades,  there  were  still  instructional  material 
problems.  The  single  largest  cause  for  poor  performance  in  all 
instructional  settings  is  the  mismatch  between  the  skills,  knowledge, 
aptitudes,  and  attitudes  a  student  brings  to  the  task  and  what  the  designer 
of  the  training  materials  assumes  the  student  brings. 


CUES 


Questionnaire  respondents  felt  their  materials  were  well  presented  with 
clear  objectives  and  maps  of  what  the  students  are  to  do.  Instructors  are 
perceived  as  expert,  with  those  in  Cl  courses  lecturing  more,  better  holding 
the  students'  attention,  and  spending  more  time  helping  individual  students 
than  in  other  courses.  Analysis  of  the  structured  interviews  revealed 
instructional  materials  need  more  clear  directions,  variety  in  presentation, 
high-quality  graphics,  and  guidelines  to  instructors  on  their  proper  use. 

PARTICIPATION 

Questionnaire  respondents  in  all  levels  of  II  feel  that  students  spend 
much  of  their  day  in  constructive  activity.  Those  in  conventional  classes 
most  easily  see  the  importance  of  that  which  they  are  to  learn,  however,  and 
are  most  likely  to  put  in  time  beyond  the  scheduled  workday.  Those  in  Cl 
courses  also  feel  the  lesson  materials  and  instructors  are  more  likely  to 
hold  student  interest,  keep  them  from  daydreaming,  manifest  appropriate 
teaching  methods,  and  elicit  more  practice  than  do  those  in  other  courses. 
The  time  between  presentation  and  practice  was  seen  as  shortest  in  the  SP 
courses.  The  instructional  materials  are  weak  in  distributed  practice  and 
in  providing  students  opportunity  to  practice  newly-acquired  job  skills  in  a 
simulated  or  model  work  environment. 

REINFORCEMENT 

Questionnaire  respondents  in  Cl  courses  are  seen  as  more  enthusiastic, 
receiving  more  recognition  for  their  efforts,  and  more  likely  to  work  with 
and  for  students  than  do  those  in  SP  courses.  Students  in  Cl  courses  are 
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most  likely  to  see  the  importance  of  their  learning  for  a  military  career, 
and  are  most  likely  to  recommend  their  courses  to  other  students. 
Instructional  materials  were  found  to  have  little  built-in  reinforcement. 
This  could  be  due  to  the  absence  of  the  topic  from  NAVEDTRA  110A  and 
previous  guides. 

FEEDBACK 

Questionnaire  respondents  feel  that  there  is  more  testing  occurring  in 
the  SP  courses  than  Cl  and  MIX,  but  that  there  is  more  instructor 
availability  for  feedback  following  testing  in  the  Cl  courses. 
Instructional  materials  were  adequate  in  providing  feedback;  however, 
reliability  and  validity  of  the  feedback  mechanism  was  not  available  for 
study. 

CORRECTIVES 

Questionnaire  respondents  feel  that  SP  and  MIX  students  are  more  likely 
to  take  corrective  activity  within  the  classroom  or  learning  center  and  less 
likely  to  be  set  back  to  a  later  class  than  those  in  Cl.  The  availability 
of  restudy  materials  following  failure  of  an  examination  is  perceived  as 
being  high  in  all  levels  of  II,  but  highest  in  the  mixed  II  courses. 
Instructional  materials  generally  build  corrective  procedures  into  classroom 
and  learning  center  practice,  but  data  from  the  interviews  revealed  little 
use  of  peer  tutoring  or  instructor  aides  in  this  phase  of  instruction. 

COST  OF  INSTRUCTION 

The  study  results  indicate  that  the  direct  costs  of  self-paced  courses 
are  approximately  one-third  to  one-half  the  costs  of  conventional 
instruction  depending  on  the  manner  in  which  costs  are  calculated.  This, 
however,  does  not  include  curriculum  development  costs.  An  earlier  report 
found  the  cost  of  graduates  higher  in  SP  than  in  Cl  (Dickinson  and  Swope, 
1981).  If  both  sets  of  data  are  to  be  believed,  they  carry  different 
implications  for  CNET  policy.  If  average  cost  per  graduate  is  indeed  higher 
for  SP  then  this  should  certainly  affect  decisions  about  undertaking  the 
conversion  of  existing  Cl  or  new  instruction  to  self-pacing.  If,  however, 
the  direct  cost  of  delivery  is  lower  for  courses  already  individualized  then 
decisions  can  be  made  relative  to  the  continuance  of  such  courses.  In  both 
these  instances  equal  effectiveness  under  Cl  or  SP  is  assumed. 
Unfortunately,  these  and  similar  analyses  were  peripheral  issues  to  the  main 
objectives  of  the  studies  in  which  they  were  embedded.  If  these  results  can 
be  validated  through  more  detailed  analysis  of  existing  cost  data,  they 
would  provide  the  basis  for  policy  regarding  the  use  of  instructional 
strategies. 
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SECTION  V 

CONCLUSIONS  AND  RECOMMENDATIONS 

Conclusions  regarding  differences  in  instructional  practices  in  self- 
paced,  computer-managed,  mixed  and  conventional  courses  as  defined  in  NITRAS 
are  provided,  here,  together  with  recommendations  for  improving  the 
management  and  conduct  of  those  courses  and  for  modifying  NAVEDTRA  110 
(series)  and  related  instructions. 

CONCLUSIONS 

1.  Individualized  instruction,  defined  as  including  (1)  release  of 
time  constraints,  (2)  choice  of  instructional  media,  and  (3)  adjustment  to 
skill  levels  of  the  learners,  rarely  occurs  in  the  NAVEDTRACOM.  Most 
courses  categorized  as  "B,"  "P,"  or  "C"  in  NITRAS  contain  some  II,  primarily 
release  of  time  constraints  (self-pacing).  Other  aspects  of  II  are  present 
in  these  courses  in  varying,  but  in  insufficient  degrees,  to  correctly 
categorize  the  courses  as  "individualized  instruction." 

2.  In  the  courses  examined,  quality  of  instruction  elements  (cues, 

participation,  reinforcement)  were  perceived  by  questionnaire  respondents  to 
occur  with  more  frequency  in  Cl  than  in  MIX  or  SP  courses.  This  may  be  due 
to  the  lower  student/instructor  ratios  in  Cl  courses  allowing  greater 

opportunity  for  instructors  to  compensate  for  curricular  deficiencies.  In 
no  form  of  instruction  examined,  however,  were  any  of  the  elements  present 
in  the  degree  necessary  to  qualify  as  II.  Although  the  attitudes  and 

perceptions  of  students,  instructors,  and  supervisors  were  positive  toward 

all  aspects  of  instruction,  regardless  of  method,  examination  of  materials 
and  visits  to  classrooms  and  learning  centers  failed  to  validate  this 

optimistic  outlook. 

3.  Both  student  and  nonstudent  direct  costs  for  the  SP  courses  in 

this  study  were  36-58  percent  of  those  required  for  the  Cl  courses. 

4.  Self-paced  instruction  is  heavily  dependent  on  written 

instructional  materials.  Consequently,  the  proper  design  and  use  of  these 
materials  is  even  more  crucial  than  for  Cl  courses.  Self-paced  materials 
examined  in  this  study  were  superior  to  those  in  Cl  and  MIX  courses; 
however,  on  an  absolute  scale  of  adequacy,  instructional  materials  were 

found  to  be  deficient  in  all  three  types  of  courses.  Primary  areas  of 
deficiency  in  SP  materials  include  orientation  to  the  course,  teaching  of 
procedures,  use  of  memory-aiding  techniques,  inadequate  opportunities  for 

distributed  practice,  and  limited  use  of  performance  testing. 

5.  Despite  the  mutually  exclusive  labels  applied  to  the  courses  in 

this  study  ("P,"  "C,"  and  "B")  most  courses  employed  a  mix  of  instructional 
strategies  to  meet  objectives;  i.e.,  aspects  of  both  II  and  Cl  were  used  in 
most  courses.  This  approach  appears  to  represent  a  pragmatic  philosophy 
within  these  courses  of  using  instructional  practices  which  match  learning 
tasks  and  that  a  single  instructional  strategy  will  probably  not  be  suitable 
for  all  tasks  within  a  given  course. 
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6.  Instructional  strategy  (II  vs  Cl)  is  not  the  determining  factor  in 
overall  course  effectiveness  of  current  Navy  courses.  Rather,  effectiveness 
is  mediated  by  the  extent  to  which  good  instructional  practices  are  used 
within  the  courses  and  ultimately  determined  by  the  degree  to  which  proper 
instructional  objectives  have  been  defined  and  met.  Good  instructional 
practices  can  be  employed  within  various  instructional  strategies.  Since  it 
is  possible  for  properly  executed  II  and  Cl  to  be  equally  effective,  a 
choice  of  strategy  should  rest  primarily  on  the  relative  cost  efficiencies 
of  the  two  approaches. 

RECOMMENDATIONS 

1.  Revise  NAVEDTRA  110  (series)  and  related  directives  to  include  the 
following  guidance: 

a.  Use  pretests  to  determine  if  a  student  meets  the  entering 
requirements  and  to  diagnose  specific  deficiencies. 

(1)  Expand  paragraph  2. 2. 4.1,  Pretests,  to  include  the  use 
of  pretests  to  determine  that  students  have  required  entry  behavior;  i.e., 
math  skills,  typing  speed  and  accuracy,  before  allowing  them  to  enter  the 
main  body  of  the  course. 

b.  Use  appropriately-designed  training  materials  to  remediate 
those  students  who  do  not  meet  criterion  on  pretested  entry  skills. 

(1)  Expand  the  Student  Remediation  Guide  for  Use  by  Learning 
Center  Instructors,  adding  corrective  procedures  for  remediating  required 
entry  skills. 

(2)  Create  entry  skills  corrective  modules  for  student  use. 
These  modules  should  teach  the  math  skills  and  other  specific  entry  level 
skills  required  but  not  taught  in  the  main  course.  Use  variations  on  the 
corrective  math  module  of  the  BE&E  school  as  a  model. 

c.  Direct  authors  to  write  module  booklets  according  to 
guidelines  on  readability  and  comprehensibility,  with  the  goal  of  making  it 
easier  for  targeted  students  to  understand  the  text  that  they  read. 

(1)  Expand  paragraph  3. 6. 2.1  to  include  guidelines  for 
writers  to:  (a)  use  controlled  vocabulary  (both  common  and  technical 
specialized  words)  which  can  be  based  on  carefully  developed  lists  (see 
Kincaid,  et  al.,  1980),  (b)  avoid  words  and  phrases  designated  as  awkward, 
(c)  avoid  awkward  or  difficult  sentences,  and  (d)  write  at  a  readability 
grade  level  matched  to  students',  reading  ability. 

(2)  Ensure  that  the  checks  are  automatically  made  by  using 
the  Computer  Readability  Editing  System  (Kincaid,  et  al.,  1980).  The 
development  and  use  of  readability  and  comprehensibility  guidelines  is 
required  by  OPNAVINST  1510.11,  Enlisted  Fundamental  Skills  Traininq,  of  19 
August  1982. 
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d.  Provide  students  with  a  clear  description  of  what  they  will 
be  able  to  do  at  the  completion  of  the  course  and  how  the  course  is 
organized  to  help  him  achieve  this  goal.  (This  is  being  accomplished  within 
conventional  group-paced  courses,  but  not  in  the  self-paced  courses.) 

(1)  Provide  each  student  with  a  course  schedule  containing  a 
sequence  of  lesson  titles,  tests,  and  laboratory  sessions,  with  an  estimated 
completion  time  (range)  for  each  entry. 

(2)  Provide  students  with  a  copy  of  the  Student  Profile, 
describing  in  simple  terms  the  skills  they  will  have  at  the  end  of  the 
course. 


e.  Provide  directions  within  NAVEDTRA  110  (series)  instructions 

for: 

(1)  guiding  training  systems  designers  in  identifying 
learning  tasks  best  accomplished  by  group  instruction 

(2)  incorporating  Outline  of  Instruction/Instructor  Activity 
pages  in  the  Learning  Center  Instructor  Guide  to  support  instructor  led 
lectures,  discussions,  and  demonstrations  for  selected  topics 

(3)  including  information  in  the  Learning  Center  Instructors 
Guide  on  how  to  schedule  students  who  advance  to  appropriate  zones  in  the 
curriculum  to  take  part  in  group-paced  activities 

(4)  guiding  training  systems,  designers  in  the  layout  of 
classrooms  supporting  both  self-paced  and  group-paced  instruction,  and 
incorporating  these  layouts  in  the  Instructional  Management  Plan. 

f.  Use  text-graphics  pages  to  convey  visual  information  that 
cannot  be  efficiently  presented  with  words.  These  highly  illustrated  pages 
are  especially  useful  in  aiding  the  student  in  locating  components  on  a 
piece  of  equipment,  placing  switches  and  levers  in  a  prescribed  position, 
and  recognizing  signals  or  system  responses. 

(1)  Make  use  of  text-graphics  pages  in  teaching  procedures 
or  in  presenting  procedures  to  be  followed  in  laboratory  exercises  involving 
the  operation  or  maintenance  of  equipment.  Use  the  TAEG  format  model  for 
procedure  learning  (Braby,  Hamel,  and  Smode,  1982). 

g.  Provide  guidelines  to  instructors  on  how  and  when  to  conduct 
demonstrations.  Improperly  handled,  demonstrations  waste  time  and  introduce 
confusion  in  students'  minds.  Demonstrations  are  an  important  mode  of 
training  in  group-paced  and  "mixed"  instruction  and  should  be  handled 
ski  1 Iful ly. 

(1)  Give  directions  on  how  to  conduct  a  demonstration  within 
that  section  providing  guidance  to  instructional  systems  designers  on 
creating  Instructor  Guides. 
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(2)  Require  that  an  Outline  of  Instruction/Instructor 
Activity  Page  be  created  for  each  major  demonstration.  In  the  Instructor 
Activities  column,  of  this  page  spell  out  exactly  what  the  instructor  is  to 
do  in  this  specific  demonstration. 

h.  Provide  detailed  guidelines  to  instructional  systems 
designers  on  how  to  identify  when  distributed  practice  is  needed,  and  how  to 
create  materials  to  support  distributed  practice. 

(1)  Direct  instructional  systems  designers  to  build 
distributed  practice  exercises  for  that  subject  matter  not  normally  used  in 
lessons  subsequent  to  its  initial  presentation  and  add  this  guidance  to 
NAVEDTRA  110  (series)  under  the  heading  "reduce  forgetting  by  providing 
periodic  opportunity  to  recall  and  practice  infrequently  used  material."  . 

i.  Direct  instructional  systems  designers  to  provide  each 
student  with  the  opportunity  to  practice  the  newly  acquired  job  skills  in  a 
simulated  or  model  work  environment.  This  should  be  the  final  phase  of  job 
training,  and  students  should  not  be  allowed  to  leave  until  certified  that 
they  can  perform  the  work  described  in  the  student  profile. 

(1)  Modify  section  3.6.2,  Instructional  Materials  for  Self- 
paced  Courses,  and  section  3.6.3,  Learning  Center  Instructor  (LCI)  Guide,  of 
NAVEDTRA  110  (series)  to  include  directions  for  designing  exercises  in 
simulated  work  environments,  and  guidelines  to  instructors  and  students  for 
carrying  out  these  exercises.  Sample  materials  should  be  included. 

(2)  Modify  paragraph  3. 6. 4. 5,  Final  Comprehensive  Test,  of 
NAVEDTRA  110  (series)  to  specifically  state  which  part  of  the  comprehensive 
test  will,  when  possible,  be  a  job-like  performance  test  in  a  simulated  or 
model  work  environment,  and  that  students  not  be  certified  until  they  pass 
this  performance  test. 

j.  More  alternative  methods  and  materials  need  to  be  developed 
for  LCIs  to  prescribe  for  students  failing  to  show  mastery  under  the  primary 
method  of  instruction  in  a  course.  Such  "correctives"  should  include 
sound/slide  packages,  books,  and  articles  keyed  to  particular  areas  of 
student  difficulty  in  the  course,  microcomputer-based  CAI  diskettes, 
counseling  by  subject-matter  experts,  assignment  of  advanced  students  as 
tutors,  and  a  longer  school  day. 

k.  Provide  instructors  and  supervisors  in  prerequisite  courses 
with  information  concerning  former  students'  performance  in  upper-level 
courses. 

l.  Introduce  more  instructor  accountability  for  students' 
academic  success  into  the  training  system.  Make  instructors'  names  a 
permanent  part  of  every  student's  record. 

2.  Compare  the  effectiveness/efficiency  of  the  35  Navy  "P,"  "C,"  and 
"B".  courses  examined  in  this  study  using  course  specific  data  from  the  TAEG 
incremental  costing  model  and  the  CNET  training  appraisal  system.  Identify 
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and  correct  course  deficiencies  in  accordance  with  NAVEDTRA  110  (series)  and 
revisions  as  suggested  in  recommendation  1. 

3.  Conduct  a.  controlled  study  of  the  relative  effectiveness  and 

efficiency  of  the  conventional  and  individualized  instruction  formats. 
Select  a  moderate  throughput  course  with  suitable  curriculum  for  the 

development  of  an  idealized  program  of  instruction  under  either  of  the 
formats.  Ensure  that  other  aspects  of  course  management  are  suitable  for 
the  assessment  of  efficiencies.  Use  study  outcomes  to  assist  with  the 

development  of  policy  regarding  the  use  of  Cl  and  II  in  NAVEDTRACOM  courses. 

4.  Use  the  projected  life-cycle  cost  as  a  major  factor  in  choosing 

the  main  instructional  strategy  (Cl  vs  II)  for  a  new  course.  When  choosing 
whether  to  change  the  instructional  strategy  of  an  ongoing  course,  use 
incremental  costing  techniques. 
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APPENDIX  A 

COURSE  DESCRIPTION  FORM  WITH  SUMMARY  STATISTICS 


This  is  the  questionnaire  sent  to  course  managers 
in  the  initial  part  of  the  study.  The  response  categories 
are  filled  in  with  summary  statistics  from  the  62  courses 
shown  in  table  2.  All  numbers  are  frequencies,  except 
where  labelled  as  mean  (M)  or  standard  deviations  (SO). 
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COURSE  DESCRIPTION  FORM 


The  Training  Analysis  and  Evaluation  Group  (TAEG)  is  continuing  its 
study  of  variations  in  individualized  instruction  for  the  Chief  of  Naval 
Education  and  Training  (CNET).  In  the  present  phase  of  the  project,  it  is 
necessary  to  obtain  a  description  of  current  practices.  Accurate  responses 
to  the  questions  that  follow  will  be  useful  in  understanding  and  improving 
the  learning  environment  of  future  Navy  men  and  women. 

Please  have  the  persons  most  familiar  with  your  course  fill  out  the 
questionnaire.  Describe  the  course  as  it  presently  is  being  taught, 
regardless  of  your  plans  to  revise  or  change  it.  Try  to  be  as  objective  and 
factual  as  possible  in  your  answers,  even  when  the  question  seems  to  require 
a  subjective  judgment.  It  shouldn't  take  much  more  than  a  half-hour. 
Completed  forms  should  be  returned  to: 

Director 

Training  Analysis  and  Evaluation  Group 

Department  of  the  Navy 

Orlando,  FL  32813 

If  there  are  any  questions  concerning  the  project,  please  contact: 

Richard  M.  Evans 

Training  Analysis  and  Evaluation  Group 

Autovon  791-5673 


(Title  of  course  ) 

(Catalog  number) 

(Location) 

(CDP  number) 

(CIN) 

(Type) 

(Name  and  rank  of 

person  responding) 

(Autovon  number) 

(Job  title  of  person  responding) 
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1. 


2. 


3. 


4. 


5. 


6. 


7. 


8. 


9. 


10. 

11. 

12. 

13. 


Do  you  record  contact  hours  as  well  as  calendar  days  to 
completion  for  individual  students?  (Circle) . 

If  Yes,  where  are  these  data  maintained? 


yes  no 
(34  28) 


(  6  )  Students'  self-records. 

j  Master  record  files  in  the  training  office. 

(6 l)  Hard  cards  in  each  classroom. 

QD  In  a  student  academic  record  book. 

(  Tfjj  In  the  computer  memory. 

( 4  )  Other  (please  list): 

(44-1) 

(10-2) 

Is  this  course  in  shifts?  If  yes,  how  many  per  day  ?.  .  .  (  5-3) 

How  many  hours  in  a  typical  class  day  in  this  course?.  .  .  (M=7.38) 

( SD= . 82) 

How  many  learning  minutes  are  there  in  a  typical  class 

hour? .  ( M=  51 .31) 

( SD=5 .85) 


Is  break  time  included  in  your  recorded  class  hours?  .  .  .  yes  no 

(39  21) 


Can  individual  students  determine  when  breaks  will  occur?,  yes  no 

(49  13) 

Does  remediation  time  add  to  the  recorded  class 

hours  if  it  occurs  over  and  above  normal  class  time?  .  .  .  yes  no 

(23  38) 

What  is  the  average  number  of  working  days  to 

completion  of  this  course? . (M=34.89) 

(SD=29.S0) 


How  many  average  working  days  do  your  faster  students 

take  to  complete? . (M=26. 56) 

(SD=24.43) 

How  many  average  working  days  do  your  slower  students 

take  to  finish? . (M=3_6.4_9) 

S0^T07 


How  many  modules  (or  lessons)  are  there  in  this  course?.  .(M=19.00) 

( SD=1 9 .20) 


How  many  days  do  students  generally  spend  until  attain¬ 
ing  mastery  on  each  module? . (M=2 .31) 

(SP04) 

After  how  many  failed  examinations  would  a  student  in  this 
course  be  (list  the  number  of  failures  next  to  any  action): 

(M=l .00)  Given  within-class  remediation  or  corrective  activity. 

(M=l .30)  Given  outside-class  remediation  or  corrective  activity. 
(M=2.1$)  Sent  to  an  academic  review  board. 

(M=0.^)  Other  (please  list): 
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14.  What  is  the  criterion  for  mastery  in  each  lesson  or  module? 

. (M=80.66)  percent 

(SPTOS) 

15.  About  how  many  students  achieve  criterion  on 

the  first  attempt  in  most  modules  or  lessons?.  .  .  . (M-72.65)  percent 

( SD=32 .61) 

16.  In  general,  do  you  fee!  that  the  recommendations 
made  by  instructors  to  academic  review  boards,  with 
regard  to  dropping  or  setting  back  students,  are 

followed?  .  Agree  Disagree 

(45  4) 

17.  What  is  your  estimate  of  the  percent  of  students  sent  to 

academic  review  boards  who  are  set  back? . (M=26 ,82)  percent 

( S  D=3  7.35) 

18.  Estimate  of  the  percent  of  students  sent  to  academic 

review  boards  who  are  dropped  from  school  . (M-24.58)  percent 

(SD=33.49) 


19.  Does  this  course  require  a  comprehensive 

end-of -course  examination? . yes  no 

(40  21) 

20.  Can  students  fail  their  course  because  of  a  low  compre¬ 
hensive  exam  score? . yes  no 

(32  23) 


21.  For  the  following  instructional  management  functions  in 
the  classroom  or  learning  center,  indicate  (by  a  check) 
iiow  it  is  performed: 


Instructor 

Computer 

Both 

Assignment  of  learning  material 

(46 

4 

8L 

Composing  exams 

(48 

5 

2) 

Administering  exams 

(48 

3 

5) 

Scoring  exams 

(42 

10 

4) 

Prescribing  remediation  or 
correctives 

(46 

2 

10) 

Record  keeping 

(45 

0 

13) 
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22.  For  the  following  instructional  management  functions  in  the 
laboratory,  indicate  (by  a  check)  how  it  is  performed: 


Instructor 

Computer 

Both 

Assignment  of  learning  material 

(43 

4 

4) 

Composing  exams 

 (43 

1 

4) 

Administering  exams 

(44 

2 

4) 

Scoring  exams 

(44 

2 

4) 

Prescribing  remediation  or 
correcti ves 

(45 

1 

-A)   

Record  keeping 

(40 

2 

9) 

23.  Estimate  the  percent  time  that  each  of  the  below  is  present  in  this 
course  (must  add  up  to  100) : 

(M-16.03;SD=21.84)  Lecture. 

(  6.00:  11.18)  Discussion. 

(  5.82;  10.14)  Classroom  demonstration. 

(  35.76;  34.1)6 )  Self-study  of  reading  materials  in  the  classroom. 

(  2.44;  5.74)  Tutoring. 

(  £.21:  4!2I)  Films  or  television  tapes. 

(  29.90;  24.82)  Laboratory  exercises  or  demonstrations. 

(  1.3l;  3. 73)  Other  (please  list): 

24.  List  the  different  ways  this  course  has  to  teach  the  same 

objectives  (such  as  the  summary,  programmed  instruction,  and 
narrative  mentioned  in  NAVEDTRA  110)?  .  ______ 

(M=2.08;  SD=1 .41) 


25.  Estimate  the  percent  time  spent  in  classroom  self-study  that  is 
devoted  to  reading  the  following  types  of  materials  (must  add 
up  to  100) : 

(M=14.35;  SD=25.27)  Student  guides. 

(  36.61;  31.88)  Modules  with  summary,  narrative,  and  programmed  instruction 

(such  as  per  NAVEDTRA  110). 

(  2.60;  12.92)  Other  programmed  instruction. 

(  15.48;  24.41)  Handouts  prepared  by  school . 

(  8.79;  19.38)  Equipment  technical  manuals  prepared  by  contractors. 

(  6.26;  1).53)  Other  (please  list): 

26.  How  many  instructors  are  assigned  to  your  average 

classroom  or  learning  center? . (M=1.92) 

(WF$3) 
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27. 


28. 


29. 


30. 

31. 


32. 


33. 


34. 


35. 


36. 


37. 


38. 


39. 


How  many  students  are  assigned  to  your  average 

classroom  or  learning  center?  .  .  .  . ( M=2 6 . 50 ;  50=38.25) 


How  many  aides,  plowbacks,  or  proctors  are  assigned 

per  room  above? . .  •  •  (M=0.35;  SD=0.70) 

How  many  full-time  managers  and  administrators  are  assigned 

to  this  course? . (M=2.65;  SD=2 . 66 ) 


What  percent  of  the  instructor's  day  is  spent  in 

training  of  general  military  subjects  or  discipline?  .  .  (M=3.89;  SD=6.02) 


What  percent  of  this  course  is  "self-paced"? 


(M=57.61;  SD=45. 13) 


What  percent  of  this  course  is  "lock-step"  instruction?.  (M=42.87;  SD=45 . 51 ) 


Do  you  use  ASVAB  scores  or  GCT  in  order  to  get  a 
predicted  completion  time  for  your  students  in  this 

course? . .  •  yes  no 

(16  46) 


Are  students  assigned  to  special  course  materials 

based  on  some  sort  of  aptitude  score? . .  yes  no 

(  3  59) 

What  is  the  average  number  of  working  days  holding  time 

for  students  awaiting  instruction  in  this  course?  ....  ( M=1 . 18 ;  SD=2.77) 

(46  Omit) 

Estimate  the  percent  of  working  time  spent  in  each  of  the 
following  activities  for  personnel  awaiting  instruction  (must 
add  to  100) : 


General  preparatory  skills,  such  as  math  or  reading. 
Course-related  activity. 

Military  training. 

Guard,  mess  duty,  etc. 

Other  (please  list): 


How  do  students  here  learn  how  well  or  poorly  they  are  doing? 


("Mechanically,  from  tests"  n=25) 
("From  a  personal  instructor"  n=10) 
("From  any  instructor"  n=  7) 
(  Combination  of  Above  n=15) 


List  in  order  of  importance  the 
this  course: 

(Extrinsic  incentives 
(Intrinsic  incentives 
(Combination  of  the  two 


major  incentives  students  have  for  doing  well 

n=39) 

n=20) 

n-  1) 


List  actions  available  here  for  dealing  with  unmotivated  students. 


(A  "punishment-type"  response  n=18) 
("Counsel  and  Help"  n=21) 
("Special  study  group"  n=  1) 
(Combination  of  above  n=17) 
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40.  List  in  the  order  of  importance  the  incentives  an  instructor  has  for 
teaching  well  here: 

(Extrinsic  Incentives  14) 

(Intrinsic  Incentives  41) 

41.  Does  every  student  have  one  instructor  who  is 
responsible  for  his  or  her  total  success  in  the 

training  program? . yes  no 

(25  34) 

42.  Which  one  of  the  following  ideas  best  describes  this  course? 

(29)  The  time  for  learning  is  pretty  well  fixed,  with 

students  varying  in  the  degree  of  their  mastery  of 
the  course  goals. 

(33)  The  students  vary  in  the  working  days  to  finish  the  course, 
but  the  level  of  their  mastery  is  pretty  much  the  same. 

43.  Which  one  of  the  following  best  describes  the  distribution  of 
student  achievement  in  this  course? 

(34)  A  few  students  below  average,  many  about  average,  and  a  few 
above  average. 

(24)  More  above  average  than  below  average  students. 

(  4)  More  below  average  than  above  average  students. 

44.  Check  the  one  statement  that  is  closest  to  the  philosophy  of 
instructors  concerning  student  achievement  here: 

(  9)  If  we  do  a  good  job  in  giving  the  students  what  they  need 
there  will  always  be  10-30  percent  of  the  students  who  just 
cannot  get  a  solid  grasp  of  the  subject. 

(52)  If  we  do  a  good  job  in  giving  the  students  what  they  need, 

we  can  get  almost  all  students  to  learn  almost  anything  this 
course  has  to  teach. 

45.  Which  one  of  the  following  best  describes  the  pacing  in  this  course? 

(17)  All  students  progress  through  the  subject  matter  at  the 
same  speed--that  of  an  "average"  student. 

(21)  Students  pretty  much  determine  their  own  individual 
pace  in  learning  here. 

(24)  The  instructors  generally  require  students  to  maintain  a 
pace  determined  by  estimates  of  student  abilities  and 
the  difficulty  of  the  subject  matter. 
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46.  Which  one  of  the  following  best  describes  a  typical 
instructor's  action  in  this  course? 

(38)  They  go  to  the  student  and  ask  questions  individually, 
when  the  student  appears  to  be  "stumped." 

(23)  They  wait  for  students  to  request  the  help  they 
may  need. 

47.  CNETINST  3920.18,  "Policy  on  Automation,"  endorses  automation  of  all 
education  and  training  functions.  Here,  we  would  like  to  know  how 
this  course  has  used  small  microcomputers  in  this  regard.  (Omit,  if 
you  do  not  use  microcomputers.) 


Instructional  Use 

Model  or  Type 

Nimber 

Management 

(Microcomputers) 

(6) 

Delivery 

(1) 

Testing 

(3) 

Admini stration 

(3) 

Other 

(3) 

48.  How  many  of  the  permanent  personnel  in  this  course  are  owners  of 
personal  microcomputers?  (Number  M=3.43  Type  Mode  =  Microcomputers) 

(7  Courses) 

49.  Please  indicate  the  number  of  personnel  teaching  or  supervising  this 
course  who  can  write  programs  in  such  microcomputer  languages  as  BASIC 
or  Pascal . 

Introductory  Intermediate  Advanced 

Number  at  ‘  ^ 

each  level  of  (M=4.33  M=2.50  M=1 . 17) 

programming  (9  Courses  8  Courses  6  Courses) 

50.  Please  list  any  special  uses  of  microcomputers  in  this  course  that 
might  be  useful  in  other  Navy  courses: 


Thank  you 
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APPENDIX  B 

STRUCTURED  INTERVIEW  FOR 
EVALUATING  INSTRUCTIONAL  MATERIALS 


The  percent  "yes"  responses  among  the  7  Cl, 
10  MIX,  and  20.  SP  courses  are  in  parentheses 
in  the  left  margin  as  follows: 

( Cl — MIX— SP) 
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STRUCTURED  INTERVIEW  FOR  EVALUATING 
INSTRUCTIONAL  MATERIALS 


I.  PREREQUISITES 

Are  materials  matched  to  student's  prior  learning? 


Y  N 


(0-20-50) 

0 

0 

1. 

Pretest  to  determine  if  prerequisites  are  known. 

(14-10-20) 

0 

0 

2. 

Pretest  to  determine  if  student  already  has  mastered 
lesson  materials. 

(29-40-45) 

0 

0 

3. 

Prerequisites  for  a  lesson  are  taught  in  previous  lessons. 

(14-40-55) 

0 

0 

4. 

Alternate  forms  of  material  exist  to  accommodate 
variations  in  student  aptitude. 

(14-30-55) 

0 

0 

5. 

Remedial  material  prescribed  for  missed  test  questions  in 
pretest  (for  prerequisite  knowledge),  progress  checks,  and 
formal  tests. 

(0-40-63) 

0 

0 

6. 

Material  is  comprehensible  to  the  typical 

student;  i.e.,  it  passes  the  comprehensibility  check. 

(0-10-30) 

0 

0 

7. 

Material  shows  how  lesson  is  related  to  prior  learning 
(advanced  organizing). 

(88-70-65) 

0 

0 

8. 

Material  states  how  student  will  use  this  information  on 
the  job,  to  ensure  the  student  knows  why  he  is  studying. 

II. 

CUES 

Management  documents  help  form  student  expectations  for  the  course. 

(71-90-80) 

0 

0 

9. 

Terminal  and  Enabling  Objectives  for  the  lesson  are  provided 
to  the  student. 

(43-30-10) 

0 

0 

10. 

The  student  copy  of  the  course  outline  clearly  shows 
lessons  and  how  they  relate  to  each  other. 

(43-40-65) 

0 

0 

11. 

The  student  is  given  a  clear  description  of  how  the  course 
will  be  conducted. 

Management  statements  in  lesson  give  clear  directions  on  how  to  study  the 
materials. 

(57-80-80) 

0 

0 

12. 

Overview  describes  what  lesson  is  about. 

(29-80-85) 

0 

0 

13. 

Each  lesson  presents  clear  directions  to  the  student  on 
how  to  use  the  material. 

(29-40-60) 

0 

0 

14. 

The  student  has  a  means  to  record  his  progress. 
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Training  objectives  are  clear  and  technically  complete. 

0  0  15.  The  describe  what  the  student  is  expected  to  be  able  to  do, 

and  give  conditions  and  standards. 

Information  is  organized  and  formatted  for  ease  of  use. 

Knowledge  Objectives: 


Y 

N 

(43-70-90) 

0 

0 

16. 

All  information  needed  for  objective  is  concisely  stated 
in  one  place. 

(57-60-85) 

0 

0 

17. 

All  non-essential  information  is  excluded. 

(43-70-70) 

0 

0 

18. 

Information  is  displayed  in  blocks. 

Blocks  have  names  or  headings. 

(57-80-70) 

0 

0 

19. 

Verbal  information  is  presented  with  words;  visual 
information  is  presented  with  graphics. 

Performance  Objectives: 

(43-70-40) 

0 

0 

20. 

Steps  and  describe  in  order. 

(0-10-30) 

0 

0 

21. 

Rules  are  presented  to  guide  performance. 

(29-30-35) 

0 

0 

22. 

Explicit  safety  precautions  are  presented. 

(71-90-25) 

0 

0 

23. 

Demonstrations  are  given. 

(43-50-15) 

0 

0 

24. 

Demonstrations  cover  range  of  applications. 

(43-80-20) 

0 

0 

25. 

Medium  for  demonstration  allows  students  to  stop, 
repeat,  and  skip  forward  and  backward. 

start. 

Memory 

aiding 

presentation  techniques  are  used. 

(0-10-10) 

0 

0 

26. 

Mnemonics . 

(0-0-20) 

0 

0 

27. 

Chunking. 

(0-10-30) 

0 

0 

28. 

Memorable  graphics. 

(14-20-40) 

0 

0 

29. 

Emboldened  key  words. 

III 

PARTICIPATION 

(57-70-75) 

0 

0 

30. 

Students  are  directed  to  practice. 

(29-80-80) 

0 

0 

31. 

Materials  are  provided  for  practice. 

(14-80-75) 

0 

0 

32. 

Only  information  found  in  lesson  presentation  is 
practice. 

needed 
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Y 

N 

(57-80-75) 

0 

0 

33. 

Practice  problems  are  consistent  with  objective,  test 
items  and  lesson  presentation. 

(43-70-50) 

0 

0 

34. 

Skills  presented  and  practiced  in  one  lesson  are  called 
up  and  practiced  in  a  series  of  subsequent  lessons. 

(29-40-35) 

0 

0  . 

35. 

Materials  are  provided  for  distributed  practice. 

(43-80-65) 

0 

0 

36. 

Practice  exercises  provide  knowledge  of  results. 

(29-70-45) 

0 

0 

37. 

Practice  exercises  guide  remediation. 

IV 

REINFORCEMENT 

(14-20-30) 

0 

0 

38. 

The  material  contains  reinforcing  statements  to  be  displayed 
to  the  student  after  he  has  successfully  completed  a  block 
of  instruction. 

(57-60-60) 

0 

0 

39. 

Students  get  75  percent  or  more  of  the  progress  check  and 
test  items  correct. 

0 

0 

(Mark  here  if  item  35  is  checked:  "Practice  exercises 
provide  knowledge  of  results." 

V 

FEEDBACK 

Types  of  Tests. 
Progress  Checks: 


(29-70-90) 

0 

0  40. 

Written. 

(29-60-60) 

0 

0  41. 

Performance. 

Module  Progress  Tests: 

(71-80-90) 

0 

0  42. 

Written. 

(29-90-70) 

0 

0  43. 

Performance. 

Comprehensive 

within  Course 

(14-10-40) 

0 

0  44. 

Written. 

(14-20-35) 

0 

0  45. 

Performance. 

Final 

Comprehensive  Test: 

(88-40-65) 

0 

0  46. 

Written. 

(14-60-30) 

0 

0  47. 

Performance. 
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Scope  of  Tests. 


Y 

(71-90-85)  0 

(71-80-85)  0 

(71-90-70)  0 

(No  data)  0 
VI. 

(43-50-60)  0 

(43-80-55)  0 

(71-80-75)  0 


N 

0  48.  Test  items  exist  to  measure  achievement  of  each  objective. 

Structure  of  Test  Items/Answers. 

0  49.  Test  items  are  consistent  with  learning  objectives. 

0  50.  Correct  answers  to  missed  items  are  presented  to  students. 

0  51.  Test  items  pass  the  comprehensibility  check. 

CORRECTIVES 

0  52.  Directions  on  how  to  study  material  on  missed  items  are 

provided. 

0  (Mark  here  if  item  36  is  checked:  "Practice  exercises  guide 

remediation.") 

0  (Mark  here  if  item  49  is  checked:  "Correct  answers  to 

missed  items  are  presented  to  students.") 
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COMPREHENSIBILITY  CHECK 

A.  From  each  set  of  instructional  materials,  randomly  select  3  narrative 
passages  of  about  500  words  each,  and  conduct  a  CRES  analysis  on  these  passages. 

Y  N 

0  0  Less  than  05%  awkward  or  undesirable  words. 

0  0  Less  than  10%  long  sentences. 

0  0  No  lists  in  long  sentences. 

0  0  Not  higher  than  10th  reading  grade  level  (12th  for  highly 

technical  material). 

B.  From  the  exercise  and  test  items  in  the  instructional  materials,  randomly 
select  20  questions  (10  multiple  choice;  10  T/F,  if  possible)  and  analyze  them 
usinq  a  modified  CRES  routine. 

1.  In  multiple  choice  questions: 

Y  N 

0  0  Not  more  than  25%  of  the  correct  answers  are  the  longest  answers. 

0  0  No  questions  flagged  as  having  inappropriate  answers. 

0  0  No  repetitive  words  and  phrases  in  answers. 

0  0  Double  negatives  are  not  used. 

0  0  Not  more  than  15%  of  test  item  alternatives  are  more  than  22 

words. 

0  0  Reading  grade  level  of  test  items  does  not  exceed  the  10th  grade 

(12th  for  highly  technical  material). 

2.  In  true/false  questions: 

0  0  Not  more  than  15%  of  questions  are  more  than  22  words. 

0  0  No  negative  wording  in  questions.  . 

0  0  No  complex  questions  (i.e.,  "either. . .or,"  "neither. . .nor, "  and 

"or") . 

0  0  Reading  grade  level  of  test  items  does  not  exceed  the  10th  grade 

(12th  highly  technical  material). 
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APPENDIX  C 

QUALITY  OF  INSTRUCTION  QUESTIONNAIRE 


Data  are  recorded  from  37  courses  in  final  sample: 

Group  1:  7  Conventional  instruction  courses  (Cl)  n=146 
Group  2:  10  Mixed  (5  -  90%)  Courses  (MIX)  n=210 
Group  3:  20  Self-Paced  (SP)  n=958 

Questionnaires  scored  1  to  9.  Group  mean  is  indicated  on  the 
scale  with  alpha  probabilities  from  oneway  ANOVA.  Number  total  =  1,314 
cases  from  37  courses  at  9  locations. 
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This  questionnaire  was  designed  by  the  Training  Analysis  and  Evaluation 
Group  to  provide  a  snapshot  of  feelings  concerning  current  instruction. 
Your  responses  to  the  questions  will  become  part  of  a  pool  of  responses  and 
will  not  be  traceable  to  you  after  they  are  entered  into  the  computer.  No 
student,  instructor,  or  supervisor  decisions  related  to  reward  or  promotion 
may  be  made  on  the  basis  of  your  answers  here. 


Name  of  Course  Being  Rated  ____________ ___ 

Location _ 

Check  one:  Student _  Instructor _  Supervisor _ 

How  many  lessons  (modules)  in  this  course?  _ 

For  students:  How  many  lessons  (modules)  have  you  completed? 


DIRECTIONS:  Draw  a  small  vertical  line  through  the  point  on  the  two- 

sided  scale  that  best  summarizes  your  answer  to  the  question. 
For  example,  the  response  to  the  question: 

"How  do  you  feel  today?" 

J.  Good 

conveys  the  notion  that  you  feel  pretty  good.  The  remaining 
questions  are  harder.  Please  try  to  answer  every  question. 
The  summarized  information  from  hundreds  of  these  question¬ 
naires  will  be  a  guide  for  the  improvement  of  Navy 
instruction. 


Bad 


i 
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1. 


3. 


4. 


9. 


How  often  do  you  find  course  materials  here  to  be  written  as  if 
students  have  certain  skills  or  knowledge  that  they,  in  fact,  do  not 
have? 


Seldom 


CI-M 


X-SP 


Often. 


For  most  students,  the  reading  level  of  lesson  materials  in  this 
course  is 

CI-SP-MIX 

Difficult  1  .  . . . — - ■ — {4- — - !  Easy. 


To  what  extent  is  review  of  previously-learned  material  a  part  of  this 
course? 

HUch  > _ SP-ip^CI _ 4  Little. 

Prior  to  enrolling  in  this  course,  most  students'  attitude  about 
school  learning  was 

mx-sp-ci 

Poor  1 - K — i - I  Good. 


Based  on  my  knowledge  of  the  average  abilities  of  students  here,  I 
would  rate  them  as 


Overachievers 


CI- 

O- 

GO 

1 

X 

1 - 1 

- » 

1  Underachievers. 


Students'  study  habits  are  improved  by  this  course 

MX-CI-SP 


■4  Little. 


Much 


Before  students  can  take  an  examination  here,  to  what  degree  must  they 
convince  their  instructor  that  they  are  ready  for  it? 


Little 


CI-SPrMX 


1 


Much , 


How  well  is  the  information  to  be  learned  in  this  course  presented  in 
the  study  materials? 


Poor  1- 


SP-CI 


MIX 


•4  Good. 


In  this  course  the  learning  objectives  seem  very 

ci-mxrSP 


Specific  J- 


Vague. 


10. 


To  what  degree  does  an  instructor  assist  in  getting  and  holding  the 
student's  attention  to  the  instructional  materials? 


CI-MIXt-SP 


High  1- 


Low. 


56 


Technical  Report  147 


To  what  degree  are  the  instructors  in  this  course  expert  in  the 
subject? 


Low  t 


-MIX 

High. 


The  course  materials  help  students  know  where  they  are  in  the  course 
and  what  they  are  to  do  next. 

MIX-SP-CI 

Agree  1  j  ||— . .  .  1  1  Disagree. 


What  proportion  of  the  instructor's  day  seems  to  be  spent  instructing 
individual  students  in  this  course? 

SP-CI-MIX 

All  of  it  1 - -  j;  i  ■» .  i  None  of  it. 


What  proportion  of  the  instructor's  day  is  seemed  to  be  spent  giving 
directions  or  information  to  students? 

MIX-SP-CI 

Little  1 - j-| — | - 1  Much. 

It  seems  as  if  most  of  the  information  in  this  course  comes  bv 

SP-MIX  Cl  y 

Printed  materials  1 — (-4 - 1 - (  Lecture. 

or  demonstration  1 


Compared  to  other  courses,  the  amount  of  time  between  presentation  of 
a  lesson  and  the  practice  of  its  skills  here  seems 

CItMIX-SP  ■ 

Long  i  - - J-j — j - [  Short. 

The  lesson  materials  clearly  describe  why  it  is  important  that 
students  learn  the  information  presented  in  the  lesson. 


Agree 


CI- 


IX-SP 


^  Disagree. 


In  this  course  what  amount  of  the  student's  day  is  spent  in 
constructive  use  of  time  in  course  activity? 


ci-S 

p-mx 

(1  1 

H  Much. 

To  what  degree  is  the  waiting  time  between  arrival  on  base  and  the 
start  of  this  course  used  in  course-preparatory  activity7 

u.  u  ,  MIX-QIfSP 

High  1 - — - j|-j - 1  Low. 

How  much  study  time  do  students  usually  spend  beyond  the  scheduled 
academic  workday? 

CJ-MIX-SP 

Much  1 - -f- -| - 1  Little. 

The  course  lesson  materials  here  hold  students'  interest  for 

u.  *  ,  SP-MIX-CI 

Minutes  h- -  - 


-4  Hours. 
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22. 


23. 


24. 


25. 


26. 


To  what  degree  do  the  teaching  methods  used  in  this  course  seem 
appropriate? 

CI-MIX-SP 

High  1 - 1— [[— - 1  Low. 

How  easy  is  it  for  a  student  to  daydream  in  this  course? 

ci  $pnmx 

Hard  I - r - — ■“■41 .  1  Easy. 


Generally,  the  kind  of  pacing  that  most  students  do  best  under  is 

c,i  mx-sp 

Instructor-pacing  ! - - - — A - 1  Self-pacing. 


To  what  degree  can  students  here  progress  at  their  own  rate?- 

CI  F1IX.-SP 

Low  1 - -|- - - — - 1  High. 

To  what  degree  does  an  instructor  determine  the  rate  students  progress 
through  this  course? 

SP-MIX  Cl 

Low  1 - 


27. 


- !  High. 

Compared  to  other  courses,  how  much  time  do  students  here  spend 
practicing  the  things  they  have  learned? 

MX-SP-CI 

Less  1 - Hh - 1  More. 


28. 


To  what  degree  do  the  practice  and  lab  activities  of  this  course  seem 
like  work  you  may  someday  have  to  do? 

CI-MXtSP 

High  1 - - H - !  Low. 


29. 


Practice  activities  will  help  students  to  remember  course  material 
when  they  get  to  the  job. 


Agree 


c  i  -lyi 


x-sp 


Disagree. 


30. 


The  lesson  materials  provide  students  here  with  the  chance  to  practice 
using  what  they  have  learned 

MIX-.CI-SP 

Seldom  ! - Hi - 1  Often. 


31. 


32. 


How  enthusiastic  are  the  instructors  about  the  subject  of  this  course? 

CI -MI X-SP 

Much  t- . -4 - 1  Little. 


To  what  degree  do  the  instructors  seem  to  be  recognized  by  their 
supervisors  because  they  give  a  lot  of  individual  help  to  students? 


High 


CI-M 


■SP 


Low. 


33. 


Instructors  here  seem  to  believe  that  students  can  learn  the 
subject  matter. 

SP-MIX-CI 

Disagree  1 - j— Ji - 1  Agree. 
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34.  To  what  degree  do  the  instructors  here  work  to  ensure  that  all 
students  do,  in  fact,  learn? 

“  ci-mx-sp 

. . — . .  )  i_ow> 


High 


35.  Students  see  the  information  in  this  course  important  for  their  future 
civilian  career. 

MIX-CI-SP 

Disagree  I - - p— - 1  Agree. 

36.  The  incentives  for  finishing  this  course  as  quickly  as  possible  seem 

MI X-SP-CI 


Great 


-I  Small. 


37.  Graduates  see  this  course  as  important  to  their  military  career. 

sp-mix-ci 

Disagree  [- - . — - — - 


Agree. 


38.  How  highly  would  you  expect  students  to  recommend  this  course  to 
others? 

SP-MIX-CI 

Little  I - —44 — | - 1  Highly. 


39.  When  students  do  poorly  on  a  test,  to  what  degree  is  corrective  work 
allowed? 

MI X-SP-CI 

Great  I - ff- - — !  Small. 

40.  The  amount  of  testing  in  this  course  seems  to  be  too 

SP-MIX-CI 

Much  1 - 1 - 1  Little. 


41.  How  often  can  students  retake  an  examination,  should  they  fail? 


Never 


c 


MIX-.SP 


-1  Many  times. 


42.  Testing  in  this  course  occurs  almost 

Daily  tSP-ffi-C,I 


Weekly. 


43.  The  tests  here  really  measure  how  well  students  know  what  the  lesson 
materials  teach. 


Disagree  i- 


SP-CI-MIX 


a 


Agree. 


44.  To  what  degree  does  the  instructor  become  involved  with  students  after 
they  take  a  test? 

,  ,  SP-fllX-.CI 

Low  I - - — jj - 1 - 1  High. 

45.  It  seems  to  me  that  the  amount  of  time  allotted  for  relearning  after 
failure  to  pass  an  examination  is  too 

SP-CJtMIX 

— ■  '|j  j  - .  1  Much. 


Little  I- 
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50. 


46.  Compared  to  other  courses,  civilian  or  military,  how  much  do  the 
instructors  here  seem  to  be  available  to  help  students? 

,  ,  SP-CI-MIX 

Little  I - - j - jj - 1  Much. 

47.  To  what  degree  do  students  take  individual  corrective  activity  in 
classroom,  if  they  should  fail  an  important  examination? 

,  MIX-SP-CI 

High  I - pp-— - 1  Low. 

48.  In  this  course,  students  who  fall  behind  are  set  back  to  a  later 
class. 

,  SFJ-MIX  QI 

Disagree  I - - f- - 4- - 1  Agree. 


49.  When  students  miss  terms  in  practice  exercises  and  tests,  the 
availability  of  the  materials  for  restudy'is 


Good  I- 


MIX-jC. 

-SP 

1 — H 

- 1 

Poor. 


Following  failure  of  an  examination,  how  many  different  ways  of 
learning  the  material  are  there  available? 


None 


ci-mx-sp 


3 


Many. 


Thank  you. 


this 
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APPENDIX  D 

MEAN  VALUES  AND  SIGNIFICANCE  (Total  df  =  1313)  OF 
THREE  VAR 1 1  ON  QUALITY  OF  INSTRUCTION  QUESTIONNAIRE 

(These  are  the  numerical  values  for 
the  scales  plotted  in  appendix  C.) 
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TABLE  D-l .  MEAN  VALUES  AND  SIGNIFICANCE  OF  THREE 

VARIATIONS  OF  II  ACROSS  50  QI  QUESTIONS 


Question 


1 

2 

3 

4 

5 

6 

7 

8 

9 

10 
11 
12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 

23 

24 

25 

26 

27 

28 

29 

30 

31 

32 

33 

34 

35 

36 

37 

38 

39 

40 

41 

42 

43 


Cl 

MIX 

SP 

ANOVA  p 

6.38 

6.28 

6.10 

n.s.* 

6.36 

7.16 

6.57 

.01 

5.36 

5.59 

5.64 

n.s. 

6.06 

5.52 

5.57 

.05 

6.01 

5.64 

5.43 

.01 

5.81 

5.88 

5.45 

.05 

4.24 

4.75 

4.48 

n.s. 

6.99 

7.18 

6.38 

.01 

7.67 

7.17 

6.83 

.01 

7.51 

6.35 

5.99 

.01 

7.82 

7.83 

7.58 

n.s. 

6.76 

7.19 

6.84 

n.s. 

5.85 

5.34 

5.89 

.01 

6.86 

5.52 

5.89 

.01 

3.84 

7.79 

8.06 

.01 

5.37 

5.74 

6.39 

.01 

6.88 

6.42 

5.93 

.01 

6.92 

7.24 

6.99 

n.s. 

6.08 

6.05 

6.64 

.01 

5.08 

3.99 

3.98 

.01 

6.21 

5.51 

5.40 

.01 

7.00 

6.23 

6.04 

.01 

6.14 

3.41 

3.51 

.01 

3.62 

5.78 

5.83 

.01 

3.99 

7.20 

7.34 

.01 

6.64 

5.10 

4.64 

.01 

5.92 

5.19 

5.35 

.05 

7.73 

6.94 

6.46 

.01 

8.16 

7.92 

7.50 

.01 

6.26 

5.80 

6.33 

.05 

7.55 

6.72 

6.48 

.01 

5.64 

5.43 

5.18 

n.s. 

7.44 

7.54 

6.78 

.01 

7.58 

7.14 

6.79 

.01 

6.32 

6.04 

6.72 

.01 

6.31 

6.76 

6.46 

n.s. 

7.89 

7.30 

6.88 

.01 

7.12 

6.49 

6.09 

.01 

6.51 

6.74 

6.73 

n.s. 

5.38 

5.43 

5.59 

n.s. 

3.20 

5.34 

6.14 

.01 

6.78 

7.16 

7.83 

.01 

7.01 

7.08 

6.75 

n.s. 

*n.s.=not  significant 
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TABLE  D-l.  MEAN  VALUES  AND  SIGNIFICANCE  OF  THREE 

VARIABLES  II  ACROSS  50  QI  QUESTIONS  (continued) 


Question  Cl  MIX  SP  ANOVA  p 


44 

45 

46 

47 

48 

49 

50 


6.49 

5.49 

5.47 

4.60 

5.03 

4.59 

7.35 

7.38 

6.37 

6.12 

6.77 

6.54 

4.23 

6.03 

7.28 

6.95 

7.36 

6.84 

5.92 

5.92 

5.95 

.01 

.01 

.01 

.05 

.01 

.01 

n.s. 
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REVIEW  OF  THE  LITERATURE 


HISTORICAL 

The  conversion  of  Navy  instruction  to  the  individualized  learning 
center  activity  that  is  the  subject  of  this  report  has  antecedents  in  the 
beginning  of  recorded  history.  Brubacher's  A  History  of  the  Problems  of 
Education  (1947)  gives  a  number  of  precedents  for  modern  training  methods, 
among  them:  the  sorting  of  students  who  were  to  become  the  philosophers, 
warriors,  or  artisans  based  on  individual  differences  in  mental  talent, 
described  by  Plato;  the  advocacy  of  different  teaching  methods  for  differing 
student  natures  by  Qui nt i Ilian;  and  more  recently,  the  1919  Dalton, 
Massachusetts,  plan  allowing  students  to  progress  at  their  own  rate,  and 
Carleton  Washburn's  individualized  "Winnetka  Plan"  in  Illinois  also  in  1919. 

The  efficiencies  sought  today  by  accounting  for  individual  human 
differences  in  learning  were  foreseen  by  Terman  over  50  years  ago: 

If  the  differences  are  found  due  in  the  main  to 
controllable  factors  of  environment  and  training,  then, 
theoretically,  they  can  be  wiped  out  by  appropriate 
education  procedures— procedures  which  it  would  then  be 
our  duty  to  provide.  On  the  other  hand,  if  they  are 
primarily  due  to  differences  in  original  endowment,  then 
the  duty  of  the  school  is  clearly  to  provide  for 
differentiated* training  which  will  take  these  native 
differences  into  account  (Terman,  1928). 

THEORETICAL 

This  was  the  thought  that  was  particularly  attractive  to  trainers  in 
the  1960s  who  found  the  need  to  teach  a  widely  divergent  group  of 
individuals  to  successfully  pursue  highly  convergent  learning  objectives. 
Prior  to  this  time  researchers  dealt  with  individual  differences  as  the 
within-group  variance  in  their  experiments.  This  variance  was  the  "error 
term"  in  their  £_  and  t  ratios.  Good  experimental  design  sought  to  maximize 
the  group  variation  under  study,  control  the  extraneous  sources  of  variance, 
and  minimize  the  error  variance  (due  to  individual  differences  in  the 
subjects  or  learners)  (Kerlinger,  1973).  But  such  experimentation 
confounded  the  differential  effects  of  individual  differences.  Strategies 
of  schooling  based  on  this  research  made  matters  worse  by  emphasizing  the 
very  differences  that  were  such  a  problem  for  the  group-based  methods  in 
use. 


The  untangling  of  the  relationships  between  methods  of  instruction  and 
ways  of  learning  has  more  recently  occurred  in  investigations  of  "ability- 
treatment  interactions"  (Bracht,  1970;  Berliner  and  Cahen,  1973), 
Theoretically,  the  more  we  know  about  how  various  individuals  learn  various 
kinds  of  things  to  be  learned  the  better  the  learning  environment  can  be 
controlled  and  the  less  the  "error"  term  in  a  teaching  or  learning 
experiment.  The  more  treatments  an  instructional  system  has  available  to 
teach  a  given  task,  the  more  likely  it  will  have  an  efficient  and  effective 
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method  to  reach  an  individual  of  a  given  ability.  To  extend  this  logic,  if 
we  had  enough  ways  of  adapting  our  methods  of  instruction  to  a  given  ability 
student,  the  variation  of  human  achievement  should  approach  zero— that  is, 
almost  anyone  should  be  able  to  learn  most  anything. 


Fundamentals  of  such  learning  were  first  described  by  Miller  and 
Dollard  (1941)  in  an  early  theory  of  teaching.  Their  identification  of 
drive,  cues,  response,  and  reward  was  a  forerunner  to  much  of  the  later 
identification  of  the  behavioral  characteristics  of  teaching--most 
importantly  used  by  Carroll  (1963)  and  then  Bloom  (1968).  Most  of  the 
current  efforts  in  individualized  instruction  have  philosophical  roots  in 
Carroll's  "A  Model  of  School  Learning"  (1963),  which  asserts. 


Degree  of  Learning  =  f 


Time  Spent 
T i m e”  Ye” fdecf. 


"Time  spent"  is  further  differentiated  with  "time  allowed  for  learning" 
and  "time  the  learner  is  willing  to  spend,"  or  perseverance.  The  "time 
needed"  is  also  broken  down  to  that  required  because  of  the  students' 
aptitude,  "ability  to  understand  instruction,"  and  "quality  of  instruction" 
( Q I ) .  Thus,  this  model  finally  becomes. 


Degree  of  Learning 


_  f ' _~in]e_Al lowed  X  Time  willinq  to  spend  ) 
1  Time  req'cTT  AYiTYn'cfeYsTanY Y  QT.  ) 


The  development  of  these  formulas  is  expanded  in  Block  and  Burns  (1977). 


Benjamin  Bloom  (1968)  reasoned  that  by  optimum  presence  of  the 
variables  under  the  school's  control  it  would  be  possible  for  almost  all 
students  to  attain  levels  of  achievement  heretofore  attained  by  only  a  few. 
The  application  of  this  reasoning  was  called  "Learning  for  Mastery."  This 
notion  was  well  suited  for  a  military  training  environment--if  under  fixed¬ 
time  instruction  students  varied  in  the  degree  of  mastery,  and  if 
achievement  variance  around  a  given  criterion  for  mastery  is  undesirable  or 
costly,  then  Bloom’s  system  fixed  the  level  of  mastery  allowing  students  to 
vary  in  the  time  taken  to  learn.  This  idea  became  a  philosophical  basis  for 
much  of  the  so-called  "self-paced"  course  development  introduced  in  military 
instruction  during  the  1970s. 


Bloom  (1974)  has  further  reasoned  that  the  apparent  variation  in  human 
intelligence,  aptitude,  and  achievement  is  based  on  norms  reflecting  the 
elapsed  time  individuals  have  lived,  or  elapsed  time  exposed  to  schooling, 
or  elapsed  time  students  have  studied  particular  subjects.  Such  variation 
is  difficult  to  examine  in  absolute  terms.  It  is  more  useful  to  relate  it 
to  a  fixed  criterion  of  attainment  or  achievement.  Bloom  approximates  this 
variation  as  a  ratio  of  5  to  1  in  student  time  to  mastery  under  a  variety  of 
learning  conditions— said  another  way,  the  learning  time  required  by  slow 
learners  at  the  beginning  of  a  course  is  about  five  times  that  of  the  faster 
learners.  Furthermore,  when  you  eliminate  the  wasted  time  and  consider 
elapsed  time  on  task,  this  ratio  is  reduced  to  3  to  1.  Finally,  Bloom  has 
observed  that  students  nearing  completion  of  a  course  taught  by  mastery 
learning  methods  differ  only  about  1.5  to  1. 
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Anderson  (1976)  investigated  the  magnitude  and  stability  of  the 
individual  differences  in  elapsed  time  and  time  on  task  and  time  to 
criterion.  He  found  that  rate  of  learning  defined  in  both  ways  was 
alterable,  much  as  Bloom  (1976)  later  hypothesized  in  a  fully  developed 
theory  of  school  learning  (see  figure  E-l).  Prominent  in  this  model  is 
"rate  of  learning"  as  a  dependent  variable,  in  keeping  with  the  earlier 
assertion  that  in  a  given  subject  taught  with  Mastery  Learning  strategies, 
the  ratio  of'TTM  between  "slow"  and  "fast"  learners  should  diminish. 

STUDENT  INSTRUCTION  LEARNING 

CHARACTERISTICS  OUTCOMES 


Instruction 


Level  and  Type  of 
Achi evement 


Rate  of  Learning 


Affective  Outcomes 


Figure  E-l.  Major  Variables  in  the  Theory  of  School  Learning 
(after  Bloom,  1976) 


Taken  separately.  Bloom  asserts  that  Cognitive  Entry  Behaviors 
(intelligence  and  experience)  account  for  50  percent  of  the  variation  in 
school  achievement.  Affective  Entry  Behaviors  (attitudes  about  learning), 
account  for  25  percent,  and  Quality  of  Instruction  accounts  for  25  percent 
of  this  variation.  The  interactive  combination  of  these  three  major 
variables  accounts  for  about  90  percent  of  school  achievement.  Since 
cognitive  and  affective  entry  behaviors  are  nearly  inalterable  during  the 
length  of  most  courses  in  the  Navy,  we  will  deal  primarily  with  those 
elements  that  are.  Bloom's  theory  describes  six  elements  that  are  present 
in  quality  instruction.  They  were  explained  in  the  definitions  section  of 
this  paper  as  prerequisites,  cues,  participation,  reinforcement,  feedback, 
and  correctives. 

These  elements  of  quality  instruction  were  discerned  by  looking  at  the 
ideal  teaching  and  learning  situation — the  tutorial.  The  tutorial  is 
perhaps  the  easiest  situation  to  provide  for  the  foregoing  six  elements  of 
quality  instruction.  The  task  of  Navy  instructional  developers  is  to  try  to 
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design  these  ideals  of  the  one-to-one  student/instructor  situation  into  the 
many-to-one  situation. 

QI  VARIABLES 

The  kind  of  individualizing  that  takes  place  in  a  given  instructional 
context  depends  on  the  number  and  kinds  of  tools  that  are  available  in  the 
instructor's  kit.  Matlick,  et  al.  (1980)  conducted  an  extensive  review  of 
the  literature  on  the  subject  and  postulated  eight  such  controlling  factors: 
(1)  time  available,  (2)  instructional  personnel,  (3)  facilities,  (4)  manage¬ 
ment,  (5)  student  population  characteristics,  (6)  course  content/task  types, 
(7)  instructional  methods,  and  (8)  media/materials/devices.  From  these, 
they  proposed  a  four-dimensional  model  for  classifying  and  describing  models 
of  II.  The  basic  dimensions  were  objectives,  time,  proficiency,  and 
instructional  treatment--each  of  which  could  be  fixed  or  variable. 
Supposedly,  courses  could  then  be  assigned  to  one  of  the  16  categories  of 
this  2x2x2x2  model  for  comparison.  Salient  conclusions  from  their 
literature  review  were:  that  II  is  not  a  precise  term  that  there  are  many 
approaches  to  II,  and  that  there  are  few  fundamental  differences  among 
current  approaches  to  individualizing. 

A  study  of  the  effects  of  cues,  participation,  and  corrective  feedback 
in  instruction  looked  at  54  studies  involving  700  classes  (Lysakowski  and 
Walberg,  1982).  The  characteristics  of  over  14,000  students  of  these 
classes  suggested  large  and  consistent  effects  of  instructional  cues, 
participation,  and  corrective  feedback  for  learning  in  natural  settings, 
such  as  in.  a  typical  classroom.  The  authors  concluded  that  their  analysis 
strongly  confirms  the  Do! lard-Mi ller-Carroll-Bloom  theory  that  has  evolved 
during  the  past  four  decades. 

Several  studies  deal  with  the  instructor/student  (I/S)  ratio  and  its 
effect  on  achievement.  Glass  and  Smith  (1979)  conducted  a  meta-analysis  of 
80  studies  of  class  size  with  achievement,  finding  an  inverse  relationship, 
and  concluded  that  few  resources  at  the  command  of  educators  will  reliably 
produce  effects  of  the  size  found.  Glass  and  Smith  suggest,  for  example, 
that  a  student  who  might  score  at  the  63rd  percentile  when  taught 
individually,  would  score  at  about  the  37th  percentile  in  a  class  of  40 
students.  Van  Matre,  et  al.  (1981)  compared  237  Navy  BE/E  students 
undergoing  instruction  with  a  1/18  I/S  ratio  and  1/30  I/S  ratio.  They  found 
that  the  larger  class  size  may  have  detrimental  effects  on  student  time  to 
mastery,  and  it.  also  may  have  an  effect  on  instructor  administrative 
behavior.  Cohen,  Kulik,  and  Kulik  (1982)  conducted  a  meta-analysis  of  65 
school  tutoring  programs.  Here,  the  tutored  students  scored  higher  on 
examinations  and  had  more  positive  attitudes  toward  subject  matter.  Another 
recent  study  by  Kulik  and  Kulik  (1982)  looked  at  ability  grouping  as  a  way 
to  individualize  within  a  conventional  instructional  environment.  This  was 
a  meta-analysis  of  52  studies  on  the  subject,  finding  a  small  effect  size, 
but  an  effect  large  enough  to  move  a  given  student  from  the  50th  to  the  54th 
percentile  in  achievement. 

The  meta-analysis  procedure  itself  was  investigated  for  its  rigor  in 
making  conclusions  about  class  size  (Hedges  and  Stock,  1983).  Using  their 
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own  model  of  efffect  size,  in  reanalysis,  however,  did  not  suggest  any 
substantial  changes  in  the  conclusions  of  Glass  and  Smith.  The  authors 
concluded  that  there  is  higher  achievement  expectation  in  smaller  classes 
and  that  class  size  accounts  for  substantial  amounts  of  variation  in  student 
achievement. 

The  importance  of  the  effect  of  I/S  ratio  found  by  Van  Matre,  et  al . 
(1981),  is  supported  by  a  more  recent  NPRDC  study.  Johnson  and  Graham 
(1982)  kept  records  of  BE/E  and  AFUN  instructors  in  the  CMI  environment  to 
determine  the  kinds  of  activity  present,  initiating  factors,  and  time 
devoted  to  each  episode  of  the  activity.  They  found  that  while  jobs 
differed  greatly  between  and  within  courses,  the  demands  on  the  learning 
center  instructors  depended  on  the  length  and  difficulty  of  the  learning 
module  and  that  tradeoffs  between  training  effectiveness  and  the  demands  on 
the  instructor  have  not  always  been  made.  The  most  striking  observation  was 
that  in  these  computerized  courses  most  instructors  spend  their  time  in 
routine  transactions  with  the  student. 

In  a  survey  of  255  students  and  100  instructors  using  the  CMI  system  at 
San  Diego  and  Memphis,  Robinson,  et  al.  (1981)  found  that  trainees' 
attitudes  were  generally  favorable  toward  the  CMI  system  while  instructors' 
attitudes  were  not  favorable.  They  also  noted  that  the  longer  the  trainees 
stayed  in  the  Navy,  the  more  their  attitudes  became  negative. 

Morris,  Surber,  and  Bijou  (1978)  investigated  the  effect  of 

procrastination  in  75  college  students  in  an  individualized  course  with  74 
in  the  equivalent  of  an  IMI  course.  They  found  that  even  though  the  "self- 
paced"  students  procrastinated  in  the  course  activities  while  those  in  the 
instructor-paced  course  did  not,  both  groups  scored  as  well  in  achievement, 
retention,  and  attitude  measures.  They  concluded  that  although  students 

procrastinate  when  free  to  do  so,  they  proceed  evenly  through  the  course 
material  when  given  incentives  to  do  so.  Finally,  the  authors  remind  us 

that  "...students  do  not  self-pace;  they  pace  according  to  the  conditions 

that  control  pacing  behavior"  (p.  228). 

A  correlational  study  of  385  freshman  biology  students  who  were  under 
individualized  and  conventional  instruction  found  some  individual 
differences  in  learners  important.  Latta,  Dolphin,  and  Grabe  (1978)  found 
low-ability,  high  test-anxious  students  do  best  under  II— especially 
females.  They  also  found  perseverance,  measured  by  extra  lecture  attendance 
and  extra  reported  study  hours,  positively  related  to  performance  in  mastery 
learning  strategies;  but  in  traditional  instruction,  this  was  only  true  for 
males. 

Time  is  central  to  the  Carroll  model.  Centra  and  Potter  (1980) 
developed  a  model  depicting  their  notion  of  the  variables  contributing  most 
to  student  learning.  One  of  their  variables.  Time  on  Task,  was  often  cited 
in  their  review  of  the  literature.  They  note  that  it  may  be  overly 
simplistic  to  conclude  that  achievement  increases  merely  due  to  length  of 
time  in  school,  as  many  researchers  have  found.  But  Centra  and  Potter  warn 
that  this  is  not  enough— concern  must  also  exist  for  the  quality  of  time 
spent  in  the  learning  environment.  Walberg  (1982)  synthesized  three 
National  studies  of  educational  effectiveness.  Although  many  studies  show 
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no  significant  difference  between  groups,  a  "Box  Score"  of  a  percentage  of 
studies  favorable  to  the  relationship  between  time  and  learning  is  95 
percent  positive.  To  make  his  point,  Walberg  suggests  that  doubling  the 
time  students  actually  concentrate  on  study  might  double  the  amount  of 
learning. 

Block  and  Tierney  (1974)  compared  two  corrective  procedures  in  a 
Mastery  Learning  instructional  strategy:  correction  by  redirected  study  and 
correction  in  the  Bloom  method,  which  was  high  in  feedback  and  formative 
evaluation.  The  redirected  study  correction  procedure  had  no  effect  on  the 
knowledge  outcomes  of  the  upper  division  historiography  course,  but  the 
Bloom-type  increased  the  application  scores. 

A  recent  review  of  13  studies  sponsored  by  the  Army  Research  Institute 
was  concerned  with  the  retention  of  tasks  performed  within  the  operationa 
military  environment  (Hagman  and  Rose,  1983).  It  concluded  that  retention 
was  improved  by  repetition,  distibuted  practice,  and  training  tailored  for 
specific  environments.  It  also  concluded  there  were  no  detrimental  effects 
in  the  use  of  equipment  variety  in  training  when  equipment  is  similar  and 
repetitions  are  spaced,  and  that  the  use  of  mnemonics  are  not  universally 
useful . 

Thompson  (1980)  compared  an  individualized  mastery  system  of  teaching 
calculus  with  a  conventional  lecture-discussion-recitation  strategy  among 
840  Air  Force  Academy  freshmen.  He  found  indistinguishable  results  in  math 
achievement,  and  suggests  that  differences  favoring  II  in  the  literature  may 
be  due  to  the  inferiority  of  Cl,  rather  than  the  superiority  of  II,  since 
the  Cl  courses  in  his  study  were  thought  to  be  well-taught.  Thompson 
suggests  that  a  less-professional  instructor  is  needed  in  II,  which  makes  it 
more  cost-effective.  He  also  warns  that  assignment  of  Cl-conditioned 
faculty  to  II  courses  may  result  in  a  loss  of  job  satisfaction  among 
instructors. 

COSTS 


TAEG  Technical  Report  105  contrasted  the  operating  costs  of  CMI,  IMI, 
and  Cl  in  a  Navy  RM  A  course  (Corey,  1981),  showing  nearly  similar  annual 
operating  costs,  but  with  15-year  cycle  operating  costs  favoring  IMI  and 
CMI.  While  no  generalizations  were  made  beyond  the  course  studied,  the 
report  did  conclude  that  the  most  expensive  resource  in  training  is  the 
student  population,  and  that  dollars  spent  in  curriculum  development  show 
great  potential  for  payback  in  a  short  time.  The  report  also  recommended 
that  formal  economic  analysis  be  a  part  of  every  major  course  development. 

Orlansky  and  String  (1979)  studied  cost-effectiveness  of  computer- 
based  instruction  in  all  of  the  military  services.  They  concluded  that 
while  achievement  in  CMI,  CAI,  and  Cl  are  about  the  same,  that  computer- 
based  education  saves  about  30  percent  time  to  mastery  over  Cl.  A  later 
study  (Orlansky,  String,  and  Chatelier,  1982)  looked  at  examples  of  flight 
simulators,  computer-based  instruction,  and  maintenance  training  simulators 
for  cost  and  effectiveness  with  nearly  the  same  conclusions.  They  also  cite 
summary  figures  giving  some  indication  of  the  importance  of  training  to  the 
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Department  of  Defense— a  $12.8  billion  enterprise  in  FY  83.  The  time  spent 
by  students,  instructors,  and  others  in  individual  training  account  for 
about  one-fourth  of  the  military  and  civilian  man-years  in  DOD.  About  20 
percent  of  all  military  personnel  are  in  schools  as  students  or  instructors, 
and  about  76  percent  of  this  effort  is  initial  training  for  new  service 
members. 

SUMMARY 

The  various  versions  of  individualized  instruction  have  antecedents  in 
the  wisdom  of  recorded  history,  with  solid  roots  in  the  learning  theory  of 
this  half  century.  From  the  tutorial  to  the  large  group  presentation,  there 
are  definable  attributes  of  effective  instruction  that  are  not  present  in 
ineffective  instruction.  The  reconceptualization  of  school  organization 
possible  by  accepting  learning  time  as  a  variable  allows  the  planning  of 
instruction  that  can  hold  levels  of  performance  constant.  This  allows  the 
development  of  a  new  philosophy  of  training  that  can  retain  high  exit 
requirements  from  courses  that  may  not  necessarily  have  control  over  the 
ability  levels  of  students  entering  the  programs. 

Benjamin  Bloom's  theory  of  school  learning  is  seen  as  the  tacit 
philosophical  foundation  of  most  military  individualized  instruction. 
Adaptive  instruction  accounts  for  the  cognitive  and  affective  entry 
behaviors  for  a  course  by  designing  instruction  for  known  levels  of 
prerequisites.  Instructional  quality  is  determined  by  the  presence  or 
absence  of  cues,  participation,  reinforcement,  feedback,  and  correctives. 
When,  these  six  attributes  are  optimum  for  a  given  human  ability,  the 
philosophy  guiding  Learning  for  Mastery  can  assert  that  all  can  learn,  and 
all  will  learn.  The  philosophy  asserts  that  it  may  take  more  time  and 
effort  for  some  than  for  others— but  they  will  learn. 
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APPENDIX  F 

A  MODEL  OF  INDIVIDUALIZED  INSTRUCTION 
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A  MODEL  OF  INDIVIDUALIZED  INSTRUCTION 

This  appendix  attempts  to  synthesize  that  which  was  learned  from  a 
review  of  the  literature  and  that  which  was  obtained  from  site  visits  and 
questionnaires.  Figure  F-l  is  a  flow  diagram  describing  ideal 
individualized  instruction.  It  is  consonant  with  the  instructional  theories 
that  provide  the  foundation  for  mastery  learning.  It  attempts  to  synthesize 
what  was  learned  by  visiting  37  Navy  and  Marine  Corps  learning  centers  and 
classrooms.  Finally,  it  assumes  that  all  students  who  begin  the  process  can 
and  wi 1 1  learn. 

In  figure  F-l  the  instructional  module  is  the  basic  unit  of  analysis. 
Such  a  module  should  be  tailored  so  that  it  will  require  an  average  student 
1  day  to  master.  In  this  way,  oral  and  printed  formative  examinations  will 
occur  often  enough  to  keep  the  student's  attention  and  require  a  high  degree 
of  participation.  The  system  is  designed  to  keep  the  student's  mind  engaged 
with  the  subject  matter.  Students  who  already  have  the  skills  taught  in  the 
module  have  the  ability  to  challenge  the  criterion  and  move  on  to  other 
instructional  modules,  if  they  can  demonstrate  mastery. 

The  process  portrayed  in  the  model  is  self-correcting.  Students  not 
having  either  the  prerequisites  or  the  mastery  of  the  module  following 
practice  do  not  exit  until  they  have  the  criterion  skills.  External 
intervention  may  be  required  when  higher  authority  determines  a  given 
student  is  spending  too  much  time  in  a  course,  and  that  the  Navy's  training 
dollars  might  be  more  effectively  spent  on  someone  else. 

The  first  decision  point  in  the  model  involves  prerequisites.  PRQs 
importance  in  the  model  reflects  Bloom's  (1976)  assertion  that  it  accounts 
for  50  percent  of  student  variation  in  achievement.  The  basic  decisions 
concerning  the  students'  possession  of  PRQ  can  be  made  with  aptitude  and 
achievement  tests  and/or  oral  examination.  If  students  do  not  have  the 
prerequisite  cognitive  and  affective  behaviors  and  if. the  instruction  itself 
does  not  adapt  to  these  deficiencies,  they  will  probably  fail  to  learn  their 
tasks.  Thus,  the  instruction  adapts  to  the  individual  by  (1)  branching  to  a 
task  for  which  the  student  does  have  the  prerequisites  or  (2)  the 
instructional  system  itself  finding  the  materials,  instructor  or  tutor 
time,  and  any  other  resources  necessary  to  teach  these  prerequisites. 

Next,  the  cues  are  presented  to  enable  the  students  to  determine  that 
which  they  are  required  to  learn.  As  with  other  quality  of  instruction 
variables  CUE  does  not  necessarily  have  to  follow  PRQ  or  come  at  any 
particular  time.  As  a  first  approximation,  cues  are  the  terminal  learning 
objectives  for  the  module.  Later  when  information  is  being  presented,  cues 
aid  in  learning  the  discriminations  necessary  to  acquire  the  concepts  or 
other  skills  involved  in  the  module. 

Participation  (PAR)  is  the  focus  of  all  activity  within  the  model. 
Students  learn  what  they  do.  Thus,  PRQ  is  an  attempt  to  determine  the 
students'  readiness  for  doing  something.  CUEs  let  students  know  what  they 
are  to  do .  FSTT  gives  students  information  as  to  how  they  are  doing.  RNF  is 
supposed  to  strengthen  that  which  students  were  doing.  And  tOR  provides 
alternate  ways  for  students  to  learn  to  do  the  particular  learning  tasks. 
All  of  these  elements  serve  tcTkeep  the'TAlT'going. 
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Figure  F-l.  A  Model  of  Quality  Individualized  Instruction 
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After  the  prerequisites  are  assured,  it  is  the  occurrence  of 
reinforcements  that  keep  the  students'  attention  to  the  cues,  the  activity, 
the  many  forms  of  testing,  and  the  correctives.  Figure  F-l  shows  RNF 
occurring  at  almost  every  point.  With  some  students,  learning  is  its  own 
reward.  With  others,  there  is  a  need  for  profuse  administration  of 
"attaboys"  or  even  more  material  reward.  This  model  of  individualized 
instruction  asserts  the  importance  of  RNF  at  a  number  of  key  points  in  the 
act  of  instruction. 

Feedback  can  also  occur  at  different  stages  in  the  model  through 
different  media.  Usually,  FBK  is  given  by  a  test,  with  additional  feedback 
provided  by  the  instructor  following  the  test.  A  test  is  defined  as  any 
evaluation  of  a  student's  performance.  Thus,  FBK  occurs  during  and  after 
examinations,  in  discussion  with  instructors,  and  in  rap  sessions  with  other 
students  concerning  aspects  of  the  course. 

Correctives  (COR)  are  a  form  of  remedial  activity  that  are  essential  to 
ensure  that  all  students  will  learn.  When  a  student  fails  to  demonstrate 
mastery  after  following  a  particular  instructional  sequence,  it  is  incumbent 
on  both  the  instructional  materials  and  the  instructor  to  determine  the 
reason  for  failure.  Once  the  cause  of  this  failure  is  diagnosed, 
alternative  materials  are  prescribed— presumably,  these  materials  will  be 
mo^e  appropriate  to  the  student's  specific  learning  style  or  skills.  Such 
alternatives  might  be  chapters  in  textbooks  or  technical  manuals, 
sound/slide  presentations,  floppy  disks  containing  microcomputer-based  CAI, 
or  tutoring  by  an  advanced  student. 

The  model  is  intended  as  a  description  of  the  essential  functions  that 
need  to  be  performed  in  good  instruction.  It  is  not  nor  is  it  intended  to 
be  a  prescription  of  how  these  functions  are  to  be  carried  out  within  the 
prerogative  of  course  managers  or  curriculum  developers.  Adequate  guidance 
for  carrying  out  these  functions  can  be  obtained  from  the  literature  and 
especially  NAVEDTRA  110A  with  the  modifications  recommended  in  section  V. 
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